ABSTRACT

The face recognition is a fairly controversial subject right now. A system such as this can
recognize and track dangerous criminals and terrorists in a crowd, but some contend that it is
an extreme invasion of privacy. The proponents of large-scale face recognition feel that it is
a necessary evil to make our country safer. It could benefit the visually impaired and allow
them to interact more easily with the environment. Also, a computer vision-based
authentication system could be put in place to allow computer access or access to a specific
room using face recognition. Another possible application would be to integrate this
technology into an artificial intelligence system for more realistic interaction with humans.
We propose a face recognition method called the PCA (Principal Component Analysis).
Principal Component Analysis (PCA) is a statistical method under the broad title of factor
analysis. The purpose of PCA is to reduce the large dimensionality of the data space
(observed variables) to the smaller intrinsic dimensionality of feature space (independent
variables), which are needed to describe the data economically. This is the case when there is
a strong correlation between observed variables. The jobs which PCA can do are prediction,
redundancy removal, feature extraction, data compression, etc. Because PCA is a known
powerful technique which can do something in the linear domain, applications having linear
models are suitable, such as signal processing, image processing, system and control theory,
communications, etc. The main idea of using PCA for face recognition is to express the large
1-D vector of pixels constructed from 2-D face image into the compact principal components
of the feature space. This is called eigenspace projection. Eigenspace is calculated by
identifying the eigenvectors of the covariance matrix derived from a set of fingerprint images
(vectors).

1|Page
 1. INTRODUCTION

1.1. PCA

The Principal Component Analysis (PCA) is one of the most successful techniques that have
been used in image recognition and compression. PCA is a statistical method under the broad
title of factor analysis. The purpose of PCA is to reduce the large dimensionality of the data
space (observed variables) to the smaller intrinsic dimensionality of feature space
(independent variables), which are needed to describe the data economically. This is the case
when there is a strong correlation between observed variables.
The jobs which PCA can do are prediction, redundancy removal, feature extraction,
data compression, etc. Because PCA is a classical technique which can do something in the
linear domain, applications having linear models are suitable, such as signal processing,
image processing, system and control theory, communications, etc.
Face recognition has many applicable areas. Moreover, it can be categorized into face
identification, face classification, or sex determination. The most useful applications contain
crowd surveillance, video content indexing, personal identification (ex. driver’s licence), mug
shots matching, entrance security, etc. The main idea of using PCA for face recognition is to
express the large 1-D vector of pixels constructed from 2-D facial image into the compact
principal components of the feature space. This can be called eigenspace projection.
Eigenspace is calculated by identifying the eigenvectors of the covariance matrix derived
from a set of facial images (vectors).

1.2. MATHEMATICS OF PCA

A 2-D facial image can be represented as 1-D vector by concatenating each row (or column)
into a long thin vector. Let’s suppose we have M vectors of size N (= rows of image £
columns of image) representing a set of sampled images. pj’s represent the pixel values.

x i= [ p 1 … … p N ] T ; i=1 ; … … ; M (1)

The images are mean centred by subtracting the mean image from each image vector. Let m
represent the mean image.
M
m=1/ M ∑ xi (2)
i=1

And let w i be defined as mean centred image

w i= x i – m (3)

Our goal is to find a set of e i which have the largest possible projection onto each of the w i.
We wish to find a set of M orthonormal vectorse i for which the quantity
M
1
λ i= ∑ (e T w )2
M i =1 i n
(4)

2|Page
is maximized with the orthonormality constraint

e Tl e k =δ lk (5)
e λ
It has been shown that the i and i are given by the eigenvectors and eigenvalues of the
covariance matrix

C = W WT (6)

where W is a matrix composed of the column vectors w i placed side by side. The size of C is
N × N which could be enormous. For example, images of size 64 × 64 create the covariance
matrix of size 4096×4096. It is not practical to solve for the eigenvectors of C directly. A
common theorem in linear algebra states that the vectors e i and scalars λ i can be obtained by
solving for the eigenvectors and eigenvalues of the M ×M matrixW T W . Let d i and μi be the
eigenvectors and eigenvalues of W T W , respectively.

W T W di =μ i di (7)

By multiplying left to both sides by W

WW T (W d i ) = μi (Wd i ) (8)

which means that the first M - 1 eigenvectors e i and eigenvalues λ i of W W T are given by Wd i
and μirespectively. Wd ineeds to be normalized in order to be equal to e i Since we only sum up
a finite number of image vectors, M, the rank of the covariance matrix cannot exceed M - 1
(The -1 come from the subtraction of the mean vector m).

The eigenvectors corresponding to nonzero eigenvalues of the covariance matrix produce an

orthonormal basis for the subspace within which most image data can be represented with a
small amount of error. The eigenvectors are sorted from high to low according to their
corresponding eigenvalues. The eigenvector associated with the largest eigenvalue is one that
reflects the greatest variance in the image. That is, the smallest eigenvalue is associated with
the eigenvector that finds the least variance. They decrease in exponential fashion, meaning
that the roughly 90% of the total variance is contained in the first 5% to 10% of the
dimensions.
A facial image can be projected onto M ı (≪ M ) dimensions by computing

Ω=[v 1 v 2 v 3 … … .. v M ]T
1 (9)

where vi =e Ti wi . v i is the i t h coordinate of the facial image in the new space, which came to be
the principal component. The vectors e iare also images, so called, eigenimages, or eigenfaces
in our case. They can be viewed as images and indeed look like faces. So,Ω describes the
contribution of each eigenface in representing the facial image by treating the eigenfaces as a
basis set for facial images. The simplest method for determining which face class provides the
best description of an input facial image is to find the face class k that minimizes the
Euclidean distance
ϵ k=‖Ω−Ωk‖ (10)

3|Page
where Ω k is a vector describing the k t h face class. If ϵ k is less than some predefined threshold
θ ϵ, a face is classified as belonging to the class k.

2. FACE RECOGNITION

Once the eigenfaces have been computed, several types of decision can be made depending
on the application. What we call face recognition is a broad term which may be further
specified to one of following tasks:

 identification where the labels of individuals must be obtained,

 recognition of a person, where it must be decided if the individual has already been
seen,
 Categorization where the face must be assigned to a certain class.

PCA computes the basis of a space which is represented by its training vectors. These basis
vectors, actually eigenvectors, computed by PCA are in the direction of the largest variance
of the training vectors. As it has been said earlier, we call them eigenfaces. Each eigenface
can be viewed a feature. When a particular face is projected onto the face space, its vector
into the face space describes the importance of each of those features in the face. The face is
expressed in the face space by its eigenface coefficients (or weights). We can handle a large
input vector,
facial image, only by taking its small weight vector in the face space. This means that we can
reconstruct the original face with some error, since the dimensionality of the image space is
much larger than that of face space.
In this report, let’s consider face identification only. Each face in the training set is
transformed into the face space and its components are stored in memory. The face space has
to be populated with these known faces. An input face is given to the system, and then it is
projected onto the face space. The system computes its distance from all the stored faces.
However, two issues should be carefully considered:

 What if the image presented to the system is not a face?

 What if the face presented to the system has not already learned, i.e., not stored as a
known face?

The first defect is easily avoided since the first eigenface is a good face filter which can test
whether each image is highly correlated with itself. The images with a low correlation can be
rejected. Or these two issues are altogether addressed by categorizing following four different
regions:

1. Near face space and near stored face⇒ known faces

2. Near face space but not near a known face ⇒ unknown faces
3. Distant from face space and near a face class ⇒ non-faces
4. Distant from face space and not near a known class ⇒ non-faces

Since a face is well represented by the face space, its reconstruction should be similar to the
original; hence the reconstruction error will be small. Non-face images will have a large

4|Page
reconstruction error which is larger than some threshold θr . The distance ϵ kdetermines
whether the input face is near a known face.

3. REQUIREMENTS

3.1. HARDWARE REQUIREMENTS

o Camera integrated system

3.2. SOFTWARE REQUIREMENTS

o Windows OS
o Visual Studio
o Open-CV version 2.10

5|Page
 4. UNDERLYING PRINCIPLE

4.1. Main idea behind eigenfaces

- Suppose G is an N 2 ×1vector, corresponding to an N ×N face image I .

- The idea is to represent Γ ¿ mean face) into a low-dimensional space:
Φ^ - mean = w 1 u1 + w2 u 2+ … … … … .+ wk uk (K<<N2)

4.2. Computation of the Eigenfaces

Step 1: obtain face images I 1 , I 2 , I 3 , … . , I M (training faces)

(very important: the face images must be cantered and of the same size)

Step 2: represent every image I i as a vector Γ i

Step 3: compute the average face vector Ψ :
M
1
Ψ = ∑ Γi
M i=1
Step 4: subtract the mean face:

Φ i=Γ i−Ψ

Step 5: compute the covariance matrix C:

M
1 T T
C = ∑ Φn Φ n = A A ¿
M n=1
where A =[Φ 1 Φ 2 … … Φ M ] ( N 2 ×M matrix)

Step 6: compute the eigenvectors ui of A AT

The matrix A AT is very large --> not practical !!
Step 6.1: consider the matrix AT A (MxM matrix)
Step 6.2: compute the eigenvectors vi of AT A
AT A v i = μi v i
What is the relationship between us iand vi ?
AT A v i=μi v i ⇒ AA T A v i=μi A v i ⇒
CA v i =μ iA vi or μi uiwhere ui A vi
T T
Thus, A A and A A have the same eigenvalues and their eigenvectors
are related as follows: μi =A vi !!
Note 1: A AT can have up to N 2 eigenvalues and eigenvectors.
Note 2: AT A can have up to M eigenvalues and eigenvectors.

6|Page
 Note 3: The M eigenvalues of AT A (along with their corresponding
eigenvectors) correspond to the M largest eigenvalues of A AT (along
with their corresponding eigenvectors).
Step 6.3: compute the M best eigenvectors of A AT : ui=¿A vi
(important: normalize ui such that ||ui || =1)
Step 7: keep only K eigenvectors (corresponding to the K largest eigenvalues)

7|Page
 4.3. REPRESENTING FACES ONTO THIS BASIS
- Each face (minus the mean) Φ i in the training set can be represented as a linear
combination of the best K eigenvectors:
K
Φ i−mean=∑ w j u j ,( w j=uTj Φ i)
j=1
(we call the u j ’s eigenfaces)

- Each normalized training face Φ i is represented in this basis by a vector:

wi1
Ω i= … . , i=1,,2.......,M
w iK

8|Page
 4.4. FACE RECOGNITION USING EIGEN FACES
- Given an unknown face image Γ(centered and of the same size like the training
faces) follow these steps:

Step 1: normalize Γ :Φ=Γ −Ψ

Step 2: project on the eigenspace

K
^ ∑ w i u i( wi=u Ti Φ)
Φ=
i=1

Step 3: represent Φ as:

w1
Ω= …
wk
Step 4: find e r =minl‖Ω−Ωl‖

Step 5: ife r ,< T r then Γ is recognized as face l from the training set.
- The distance e r , is called distance within the face space (difs)

Comment: we can use the common Euclidean distance to compute er, however, it
has been reported that the Mahalanobis distance performs better:
k
1
‖Ω−Ωk‖=∑ λ ¿
i=1 i
(variations along all axes are treated as equally significant)

9|Page
 4.5. FACE DETECTION USING EIGENFACES
- Given an unknown image Γ
Step 1: compute
Φ=Γ −Ψ
Step 2: compute
K
^ ∑ w i u i( wi=u Ti Φ)
Φ=
i=1

Step 3: compute
e d =‖Φ−Φ ^‖
Step 4: if e d < T d, then Γ is a face.
- The distance e d is called distance from face space (dffs)

10 | P a g e
 5. TECHONOLOGY USED
5.1. MICROSOFT VISUAL C++

Visual C++ is IDE of Microsoft to develop application using C++ & its STL (Standard
Template Library). Application developed by visual C++ can be of different types such as
MFC application, win console application etc. In our case, we will be developing win console
application using OpenCV.

5.2. OPEN CV

OpenCV [OpenCV] is an open source (see http://opensource.org) computer vision library

available from http://SourceForge.net/projects/opencvlibrary. Th e library is written in C and
C++ and runs under Linux, Windows and Mac OS X. Th ere is active development on
interfaces for Python, Ruby, Matlab, and other languages.
OpenCV was designed for computational efficiency and with a strong focus on realtime
applications. OpenCV is written in optimized C and can take advantage of multicore
processors. If you desire further automatic optimization on Intel architectures [Intel], you can
buy Intel’s Integrated Performance Primitives (IPP) libraries [IPP], which consist of low-
level optimized routines in many different algorithmic areas. OpenCV automatically uses the
appropriate IPP library at runtime if that library is installed.
One of OpenCV’s goals is to provide a simple-to-use computer vision infrastructure that
helps people build fairly sophisticated vision applications quickly. The OpenCV library
contains over 500 functions that span many areas in vision, including factory product
inspection, medical imaging, security, user interface, camera calibration, stereo vision, and
robotics. Because computer vision and machine learning often go hand-in hand, OpenCV also
contains a full, general-purpose Machine Learning Library (MLL). This sub library is focused
on statistical pattern recognition and clustering. The MLL is highly useful for the vision tasks
that are at the core of OpenCV’s mission, but it is general enough to be used for any machine
learning problem.

11 | P a g e
5.2.1 OpenCV Structure and Content

OpenCV is broadly structured into fi ve main components, four of which are shown in
Figure. The CV component contains the basic image processing and higher-level computer
vision algorithms; ML is the machine learning library, which includes many statistical
classifiers and clustering tools. HighGUI contains I/O routines and functions for storing and
loading video and images, and CXCore contains the basic data structures and content.

12 | P a g e
6. USER DEFINED FUNCTIONS AND THEIR PURPOSE
6.1. printusage

This function is used to control the program from the command line user
interface. Depending upon the user input it transfers the control of program to either
training phase or testing phase or online recognition of a face from camera.

6.2. learn

This function is used for offline recognition mode where it trains from the data
given in text file and store the trained data into the file ‘facedata.xml ‘.

6.3. doPCA

This function is used to perform PCA operation on the training faces.

6.4. StoreTrainingdata

This function is used to store the recognition data as an xml file.

6.5. loadTrainingdata

This function is used to load the training data from the file 'facedata.xml'.

6.6. Findnearestneighbour

This function is used to find the most likely person based on detection. It returns
the index of image, and stores the confidence value into variable pConfidence.

6.7. loadfaceImgArray

This function is used to read the names & image of people from a text file, and
load all those images listed.

6.8. Recognizefilelist

It is used to recognize the face in each of the test images given, and compare
the results with the truth.

6.9. Getcameraframe

It is used to grab the next camera frame. It waits until the next frame is ready,
and provides direct access to it, as a result it do NOT modify the returned image or
free it. It automatically initializes the camera on the first frame.

13 | P a g e
6.10. Convertimageto grayscale

It is used to return a new image that is always greyscale, irrespective of the input
type of image. It free the returned image using cvReleaseImage() when finished.

6.11. cropImage

It returns a new image that is a cropped version of the original image.

6.12. resizeImage

It creates a new image copy that is of a desired size. Remember to free the new
image later.

6.13. convertFloatImagetoUcharImage

It gets an 8-bit equivalent of the 32-bit Float image. It returns a new image, so
remember to call cvReleaseImage() on the result.

6.14. SaveFloatImage

It stores a greyscale floating-point CvMat image into a BMP/JPG/GIF/PNG

image,since cvSaveImage() can only handle 8bit images (not 32bit float images).

6.15. detectFaceInImage

Perform face detection on the input image, using the given Haar cascade
classifier. It returns a rectangle for the detected region in the given image.

6.16. Retrainonline

It is used to re-train the new face recorded in a database without shutting

down.Depending on the number of images in the training set and number of people, it
might take 30 seconds or so.

14 | P a g e
7. Snapshots of project on running mode

Figure : It represents start of the project where user have to enter a command, according to
which it wil run further.

Figure : It represents status of the program when user has entered recognize as its command.

15 | P a g e
Figure : It represents status of the program where the program is trying to recognize a face

Figure: It represents a camera frame which has been captured by camera. After detecting a
face in the frane the program identifies it and returns name of the function and its confidence
level.

16 | P a g e
8. Future Work
There are many interesting problems that remain in the area of face recognition. One problem
is image preprocessing prior to the application of the eigenface method. It may be possible to
gain better accuracy in classification if one segments the spectrum of people into different
spaces. For example, if one was able to determine if an image was of a man or a woman, one
could use this categorization to send an image to one of two classifiers, each specifically
trained with that type of individual in mind. This would mean that there would be a set of
eigenfaces specifically for males and one specifically for females (face spaces with gender, so
to speak). Work in this area has been done by Lizama, Waldoestl and Nickolay [4], however
it would be interesting to extend it to use eigenfaces to act as the gender classifier as well. A
general face-space would be created in addition to the male and female face-spaces, with the
sole purpose of being used to classify an image as male or female.
Another area of future work is improving our neural network classifier. As mentioned in the
previously, it is possible to construct the network to take its input directly from the image
data rather from the vector that results from an images projection into face-space. Perhaps
learning the face projection function could increase the accuracy of the neural network
classifier. Additionally, more experiments are needed to see if there are other ways to tweak
the network configuration to produce better results.

17 | P a g e
9. Reference
 Learning OpenCV –Computer Vision with the openCV Library O’Reilly Publication
 Gary Bradsk1 and Andrian Kehlar

 M.A. Turk and A.P. Pentland, “Face Recognition Using Eigenfaces”, IEEE Conf. on
Computer Vision and Pattern Recognition, pp. 586-591, 1991.

 Kyungnam Kim ” Face Recognition using Principle Component Analysis”

18 | P a g e