MTE Project Report

Dated -5.11.2020

Subject -MODELLING AND SIMULATION

BY: ARYAMAN DOSAJH(2K19/SE/018) AND CHIRAG
VERMA(2K19/SE/032)

● Topic- FACIAL RECOGNITION SYSTEM USING OPEN CV

1.Acknowledgment.................................................................................

2. Abstract ........................................................................................................
3.Introduction .............................................................................................

4. OBJECTIVE OF THE PROJECT.....................................

5.Software/Hardware Requirements......................................

6.Theory of face detection classifiers .........................................

7. Theory of face recognizer..............................................

8. Recognition using python and open cv....................................................

9. Conclusion...................................................................................
10. Reference.....................................................................
Acknowledgment

First of all, I would like to express my special thanks of gratitude to our parents
as well as to my teacher prof.

Rahul Sir who gave me such an opportunity to do my first review of my project.

I would like to take this opportunity to express my profound sense of gratitude

and respect to all those who helped me throughout the duration of this project.
DTU UNIVERSITY in particular has been the source of inspiration for me.I
acknowledge the effort of those who have contributed significantly to my
project.

Abstract
A face recognition system is one of the biometric information processes, its
applicability is easier and working range is larger than others, i.e.; fingerprint,
iris scanning, signature, etc. The system uses a combination of techniques in
two topics; face detection and recognition. The face detection is performed on
live acquired images without any application field in mind. Face recognition
presents a challenging problem in the field of image analysis and computer
vision. The security of information is becoming very significant and difficult.
Security cameras are presently common in airports, Offices, University, ATM,
Bank and in any locations with a security system. Face recognition is a
biometric system used to identify or verify a person from a digital image. Face
Recognition system is used in security. Face recognition system should be able
to automatically detect a face in an image. This involves extracts its features and
then recognize it, regardless of lighting, expression, illumination, ageing,
transformations (translate, rotate and scale image) and pose, which is a difficult
task. This paper contains three sections.

Introduction to the Topic

Machine learning is a subfield of Artificial Intelligence. The trends in
advancement of AI pertains its success to the computation capabilities of the
modern day CPUs and GPUs which can run large datasets and give us results.
The smart phones of today, without an exception have a selfie camera, which
generally has a rectangular boundary around the face. The selfie camera has
certain features, like it may display the age of the person, the gender of the
person. The accuracy of the features displayed by the selfie camera depends on
several factors.

Such as light, angle, beauty mode enabled etc. The smart phone industry, uses a
pre- trained model to recognize the gender and age of the person. The pre-
trained model uses deep learning to identify features using which it predicts the
gender and age of the person based on identifying and analyzing facial features.

Objective /Problem Statement

The problem statement is to recognize the face that is feeded into the model via
webcam. The image will be feeded to the pre-trained model via a webcam. This
venture was finished with this incredible "Open Source Computer Vision
Library", the OpenCV. OpenCV was intended for computational proficiency
and with a solid spotlight on continuous applications. In this way, it's ideal for
ongoing face acknowledgment utilizing a camera.

Overview of the classifier

SOFTWARE REQUIRED:-
Operating system - windows operating system
Language - python
Open source library – openCV

HARDWARE REQUIRED:-
• Processor -pentium processor or above
• RAM -minimum 64MB primary memory
• Hard disk -minimum 1gb hard disk space
• Monitor -color monitor 16 but or above
• Webcamera
• Compact disk drive
• Keyboard and mouse

NEED OF CASCADE CLASSIFIER

• In an image, most of the image is non-face region. So it is a better idea to
have a simple method to check if a window is not a face region. If it is
not, discard it in a single shot, and don't process it again. Instead, focus
on regions where there can be a face. This way, we spend more time
checking possible face regions.
• For this they introduced the concept of Cascade of Classifiers.
• OpenCV provides a training method or pretrained models, that can be
read using the cv::CascadeClassifier::load method. The pretrained
models are located in the data folder in the OpenCV installation.
• OpenCV provides us with a class cv2.CascadeClassifier which takes as
input the training file of the (Haar/LBP) classifier we want to load and
loads it for us.
Theory of face detection classifiers
A computer program that decides whether an image is a positive image
(face image) or negative image (non-face image) is called a classifier. A
classifier is trained on hundreds of thousands of face and non-face
images to learn how to classify a new image correctly. OpenCV provides
us with two pre-trained and ready to be used for face detection
classifiers:

1. Haar Classifier
2. LBP Classifier

Both of these classifiers process images in gray scales, basically

because we don't need color information to decide if a picture has a face
or not . As these are pre-trained in OpenCV, their learned knowledge
files also come bundled with OpenCV opencv/data/.

To run a classifier, we need to load the knowledge files first, as if it had

no knowledge, just like a newly born baby (stupid babies).
Each file starts with the name of the classifier it belongs to. For example,
a Haar cascade classifier starts off as haarcascade_frontalface_alt.xml.

Each OpenCV face detection classifier has its pros and cons, but the
major differences are in accuracy and speed.

So, in case more accurate detections are required, Haar classifier is the
way to go. This bad boy is more suitable in technology such as security
systems or high-end stalking.

But the LBP classifier is faster, therefore, should be used in mobile

applications or embedded systems.
 1) HAAR Classifier

The Haar Classifier is a machine learning based approach, an algorithm

created by Paul Viola and Michael Jones; which (as mentioned before)
are trained from many many positive images (with faces) and negatives
images (without faces).

It starts by extracting Haar features from each image as shown by the

windows below:

A couple Haar-like features. Source: docs.opencv.org

Each window is placed on the picture to calculate a single feature. This

feature is a single value obtained by subtracting the sum of pixels under
the white part of the window from the sum of the pixels under the black
part of the window.

Now, all possible sizes of each window are placed on all possible
locations of each image to calculate plenty of features.

Different stages in visualization. Source: docs.opencv.org

For example, in above image, we are extracting two features. The first
one focuses on the property that the region of the eyes is often darker
than the area of the nose and cheeks. The second feature relies on the
property that the eyes are darker than the bridge of the nose.

But among all these features calculated, most of them are irrelevant. For
example, when used on the cheek, the windows become irrelevant
because none of these areas are darker or lighter than other regions on
the cheeks, all sectors here are the same.

So we promptly discard irrelevant features and keep only those relevant

with a fancy technique called Adaboost. AdaBoost is a training process
for face detection, which selects only those features known to improve
the classification (face/non-face) accuracy of our classifier.

In the end, the algorithm considers the fact that generally: most of the
region in an image is a non-face region. Considering this, it’s a better
idea to have a simple method to check if a window is a non-face region,
and if it's not, discard it right away and don’t process it again. So we can
focus mostly on the area where a face is.
 2) LBP Cascade Classifier

As any other classifier, the Local Binary Patterns, or LBP in short, also
needs to be trained on hundreds of images. LBP is a visual/texture
descriptor, and thankfully, our faces are also composed of micro visual
patterns.

So, LBP features are extracted to form a feature vector that classifies a
face from a non-face.

Each training image is divided into some blocks as shown in the picture
below.

LBP Windows (disregard the first grader drawing)

For each block, LBP looks at 9 pixels (3×3 window) at a time, and with a
particular interest in the pixel located in the center of the window.

Then, it compares the central pixel value with every neighbor's pixel
value under the 3×3 window. For each neighbor pixel that is greater than
or equal to the center pixel, it sets its value to 1, and for the others, it
sets them to 0.

After that, it reads the updated pixel values (which can be either 0 or 1)
in a clockwise order and forms a binary number. Next, it converts the
binary number into a decimal number, and that decimal number is the
new value of the center pixel. We do this for every pixel in a block.

LBP conversion to binary. Local Binary Patterns applied to Face

Detection and Recognition.

Now, after you get a list of local binary patterns, you convert each one
into a decimal number using binary to decimal conversion (as shown in
above image) and then you make a histogram of all of those decimal
values. A sample histogram looks like this:

Histogram Sample.

And this is how our Face Detector will look once we finish coding
it:
Let's start with a simple task, load our input image, convert it to grayscale
mode and then display it.

For this, we’ll need to keep at arm’s reach this very handy
function cv2.cvtColor (converting images to grayscale).

This step is necessary because many operations in OpenCV are done in

grayscale for performance reasons.

To read/load our image and convert it to grayscale, we used OpenCV’s

built in function cv2.imread(img_path) and passed our image path as the
input parameter.

Given below is a sample Gray scale image

Before we can continue with face detection, we have to load our

cascade classifier.
Now, how do we detect a face from an image using the Cascade
Classifier we just loaded?

Well, again OpenCV's Cascaded Classifier has made it simple for us as

it comes with the function detectMultiScale, which detects exactly that.
Next are some details of its options/arguments:

 detectMultiScale(image, scaleFactor, minNeighbors): This is a

general function to detect objects, in this case, it'll detect faces
since we called in the face cascade. If it finds a face, it returns a
list of positions of said face in the form “Rect(x,y,w,h).”, if not,
then returns “None”.
 Image: The first input is the grayscale image. So make sure the
image is in grayscale.
 scaleFactor: This function compensates a false perception in size
that occurs when one face appears to be bigger than the other
simply because it is closer to the camera.
 minNeighbors: This is a detection algorithm that uses a moving
window to detect objects, it does so by defining how many objects
are found near the current one before it can declare the face
found.

There are other parameters as well. These parameters need to be tuned

according to your own data.

Now that we know a straightforward way to detect faces, we could now

find the face in our test image.

Then, let's find one!

The following code will try to detect a face from the image and, if
detected, it will print the number of faces that it has found, which in our
case should be 1.

#let's detect multiscale (some images may be closer to camera than

others) images
faces = lpb_face_cascade.detectMultiScale(gray_img, scaleFactor=1.1,
minNeighbors=5);

#print the number of faces found

print('Faces found: ', len(faces))

Faces found: 1

Next, let's loop over the list of faces (rectangles) it returned and drew
those rectangles using yet another built-in OpenCV rectangle function on
our original colored image to see if it found the right faces:

#go over list of faces and draw them as rectangles on original colored
img for (x, y, w, h) in faces:
cv2.rectangle(test1, (x, y), (x+w, y+h), (0, 255, 0), 2)

Let’s display the original image to see the rectangles we just drew and
verify that detected faces are real ones and not any false positives.

#convert image to RGB and show image

plt.imshow(convertToRGB(test1))

Here till now we have completed the detection of a face, NOW its time to
match the faces and recognize.

Theory of OpenCV face recognizers

Thanks to OpenCV, coding facial recognition is now easier than ever.

There are three easy steps to computer coding facial recognition, which
are similar to the steps that our brains use for recognizing faces. These
steps are:
Data Gathering: Gather face data (face images in this case) of the
persons you want to identify.
Train the Recognizer: Feed that face data and respective names of each
face to the recognizer so that it can learn.
Recognition: Feed new faces of that people and see if the face
recognizer you just trained recognizes them.

OpenCV has three built-in face recognizers and thanks to its clean
coding, you can use any of them just by changing a single line of code.
Here are the names of those face recognizers and their OpenCV calls:

 EigenFaces – cv2.face.createEigenFaceRecognizer()
 FisherFaces – cv2.face.createFisherFaceRecognizer()

 Local Binary Patterns Histograms (LBPH) –

cv2.face.createLBPHFaceRecognizer()

Here is a summary of each one

1) Eigenfaces face recognizer

This algorithm considers the fact that not all parts of a face are equally
important or useful for face recognition. Indeed, when you look at someone,
you recognize that person by his distinct features, like the eyes, nose, cheeks or
forehead; and how they vary respect to each other.

In that sense, you are focusing on the areas of maximum change. For example,
from the eyes to the nose there is a significant change, and same applies from
the nose to the mouth. When you look at multiple faces, you compare them by
looking at these areas, because by catching the maximum variation among
faces, they help you differentiate one face from the other.

In this way, is how EigenFaces recognizer works. It looks at all the training
images of all the people as a whole and tries to extract the components which
are relevant and useful and discards the rest. These important features are
called principal components.
Below is an image showing the variance extracted from a list of faces.

EigenFaces Face Recognizer Principal Components. Source: docs.opencv.org

You can see that the useful features represent faces which receive the name of
Eigen Faces. I mean, how else do you think the algorithm got its name?

So, EigenFaces recognizer trains itself by extracting principal components, but

it also keeps a record of which ones belong to which person. Thus, whenever
you introduce a new image to the algorithm, it repeats the same process as
follows:

1. Extract the principal components from the new picture.

2. Compare those features with the list of elements stored during training.
3. Find the ones with the best match.
4. Return the ‘person’ label associated with that best match component.

In simple words, it’s a game of matching.

However, one thing to note in above image is that EigenFaces algorithm also
considers illumination as an important feature. In consequence, lights and
shadows are picked up by EigenFaces, which classifies them as representing a
‘face.'

Face recognition picks up on human things, dominated by shapes and shadows:

two eyes, a nose, a mouth.
 2) Fisherfaces face recognizer

This algorithm is an improved version of the last one. As we just saw,

EigenFaces looks at all the training faces of all the people at once and finds
principal components from all of them combined. By doing that, it doesn't focus
on the features that discriminate one individual from another. Instead, it
concentrates on the ones that represent all the faces of all the people in the
training data, as a whole.

But here’s the kicker:

Consider the lighting changes in following images:

Since EigenFaces also finds illumination as a useful component, it will find this
variation very relevant for face recognition and may discard the features of the
other people's faces, considering them less useful. In the end, the variance that
EigenFaces has extracted represents just one individual's facial features.

How to fix this issue?

We can do it by tunning EigenFaces so that it extracts useful features from the
faces of each person separately instead of extracting them from all the faces
combined. In this way, even if one person has high illumination changes, it will
not affect the other people's features extraction process.

Precisely, FisherFaces face recognizer algorithm extracts principal components

that differentiate one person from the others. In that sense, an individual's
components do not dominate (become more useful) over the others.

Below is an image of principal components using FisherFaces algorithm.

FisherFaces Face Recognizer Principal Components. Source: docs.opencv.org

You can see that the features represent faces which receive the name of Fisher
Faces. Are you noticing a theme with the names of the algorithms?

One thing to note here is that FisherFaces only prevents features of one person
from becoming dominant, but it still considers illumination changes as a useful
feature. We know that light variation is not a useful feature to extract as it is not
part of the actual face.

Then, how to get rid of this problem? Here is where our next face recognizer
comes in.

3) Local binary patterns histograms (LBPH) Face Recognizer

I wrote a detailed explanation of Local Binary Patterns Histograms in my

previous article on face detection, which I’m sure you’ve read by now. So,
here I will just give a brief overview of how it works.

We know that Eigenfaces and Fisherfaces are both affected by light and, in real
life, we can't guarantee perfect light conditions. LBPH face recognizer is an
improvement to overcome this drawback.

The idea with LBPH is not to look at the image as a whole, but instead, try to
find its local structure by comparing each pixel to the neighboring pixels.

The LBPH Face Recognizer Process

Take a 3×3 window and move it across one image. At each move (each local
part of the picture), compare the pixel at the center, with its surrounding pixels.
Denote the neighbors with intensity value less than or equal to the center pixel
by 1 and the rest by 0.

After you read these 0/1 values under the 3×3 window in a clockwise order, you
will have a binary pattern like 11100011 that is local to a particular area of the
picture. When you finish doing this on the whole image, you will have a list
of local binary patterns.

LBP conversion to binary. Source: López & Ruiz; Local Binary Patterns
applied to Face Detection and Recognition.

Now, after you get a list of local binary patterns, you convert each one into a
decimal number using binary to decimal conversion (as shown in above
image) and then you make a histogramof all of those decimal values. A sample
histogram looks like this:

Histogram Sample.
In the end, you will have one histogram for each face in the training data set.
That means that if there were 100 images in the training data set then LBPH
will extract 100 histograms after training and store them for later recognition.
Remember, the algorithm also keeps track of which histogram belongs to which
person.

Later during recognition, the process is as follows:

1. Feed a new image to the recognizer for face recognition.

2. The recognizer generates a histogram for that new picture.
3. It then compares that histogram with the histograms it already has.
4. Finally, it finds the best match and returns the person label associated
with that best match.

Below is a group of faces and their respective local binary patterns images. You
can see that the LBP faces are not affected by changes in light conditions:

Recognition using Python and OpenCV

We are going to divide the Face Recognition process in this tutorial into
three steps:

1. Prepare Training Data: Read training images for each

person/subject along with their labels, detect faces from each
image and assign each detected face an integer label of the
person it belongs.
 2. Train Face Recognizer: Train OpenCV's recognizer by feeding it the
data we prepared in step 1.
3. Prediction: Introduce some test images to face recognizer and
see if it predicts them correctly.

1) Prepare training data

The premise here is simple: The more images used in training, the
better.

Being thorough with this principle is important because it is the only way
for training a face recognizer so it can learn the different ‘faces’ of the
same person; for example: with glasses, without glasses, laughing, sad,
happy, crying, with a beard, without a beard, etc.

So, our training data consists of total two people with 12 images of each
one. All training data is inside the folder: training-data.

This folder contains one subfolder for every individual, named with the
format: sLabel (e.g. s1, s2) where the label is the integer assigned to that
person. For example, the subfolder called s1means that it contains
images for person 1.

With that in mind, the directory structure tree for training data is as
follows:

On the other hand, The foldertest-data contains images that we will use
to test our face recognition program after we have trained it successfully.
Considering that the OpenCV face recognizer only accepts labels as
integers, we need to define a mapping between integer tags and the
person’s actual name. So, below I am defining the mapping of a
person’s integer labels and their respective names

Data Preparation for Face Recognition

Perhaps you are thinking: Why are are we talking about preparing data?
Well, to know which face belongs to which person, OpenCV face
recognizer accepts information in a particular format. In fact, it receives
two vectors:

 One is the faces of all the people.

 The second is the integer labels for each face.

For example, if we had two individuals and two images for each one:

Then, the data preparation step will produce following face and label
vectors:

2) Train Face Recognizer

As we mentioned earlier, OpenCV comes equipped with three face
recognizers.
 1. EigenFaces: cv2.face.createEigenFaceRecognizer()
2. FisherFaces: cv2.face.createFisherFaceRecognizer()
3. Local Binary Patterns Histogram (LBPH):
cv2.face.LBPHFisherFaceRecognizer()

3) Prediction
This is where we get to see if our algorithm is recognizing our individual
faces or not.

We’re going to take one test image of each person, use face detection
and then pass those faces to our trained face recognizer. Then we find
out if our face recognition is successful.

Below are some utility functions that we will use for drawing bounding
box (rectangle) around the face and putting the person's name near the
face bounding box.

Conclusion
It is our opinion that research in face recognition is an exciting area for many
years to come and will keep many scientists and engineers busy. In this project
we have provided concepts of face recognition methods using Open CV. The
present project can provide a better understanding about face recognition
methods.

In the future, 2D & 3D Face Recognition and large scale applications such as e-
commerce, student ID, digital driver licenses, or even national ID is the
challenging task in face recognition & the topic is open to further research.

References
1) https://www.researchgate.net/publication/260483303_Face_Recogn
ition_Methods_Applications
2) https://www.researchgate.net/publication/262875649_Design_of_a
_Face_Recognition_System
3) https://www.superdatascience.com/blogs/opencv-face-recognition
4) https://www.superdatascience.com/blogs/opencv-face-detection/