Video Based Face Recognition

Project report submitted in partial fulfillment of

the Requirements for the
Award of the Degree of B.Tech in
Computer Science and Engineering

BY

Ujjwal Shrivastav
2012657
Vasudev Rathore
2012664
Anmol Bhardwaj
2012500
Shriyansh Sahu
2012642

Under the Guidance of

Akanksha Gupta
Assistant Professor

Department of Computer Science and Engineering

Graphic Era Deemed to be University
Dehradun-248002
2021

1
 CERTIFICATE
This is to certify that the project report entitled “Video Based Face Recognition” being
submitted by
Ujjwal Shrivastav 2012657 , Vasudev Rathore 2012664 ,
Anmol Bhardwaj 2012500 and Shriyansh Sahu 2012642

in partial fulfillment for the award of the Degree of Bachelor of Technology in Computer
Science and Engineering to the Graphic Era Deemed to be University is a record of bonafied
work carried out under my guidance and supervision.

The results embodied in this project report have not been submitted to any other University
or Institute for the award of any Degree or Diploma.

(Project Guide) Devesh Pratap Singh

Akanksha Gupta Head of the Department,
Assistant Professor Computer Science,
GEU, Dehradun GEU,
Date- 19-05-2021
Dehradun

2
 ACKNOWLEDGEMENT

Acknowledgment is the most important page for any project through this we are happy to
thank and present our gratitude to the Graphic Era Deemed to be
University for giving us the opportunity to work in this project of “Video Based Face
Recognition”. Also, we would like to show our mentor,
Akanksha Gupta (Assistant Professor, Department of Computer Science and Engineering),
our sincerest gratitude for giving us an opportunity to discover our skills and use them to
work on such an important project. This project will not only enhance our technical skills but
also helped us to learn about trending technological advancements in the field of Computer
Vision.

3
 ABSTRACT

During the previous quite a while, face Reocognition in video

has gotten huge consideration. Not just the wide scope of business and law implementation
applications, yet in addition the accessibility of plausible advancements following a very
long while of exploration contributes to the pattern. Albeit current face acknowledgment
frameworks have reached a certain degree of development, their advancement is as yet
restricted by the conditions achieved by numerous genuine applications.

For instance, acknowledgment pictures of video succession procured in an

open climate with changes in light or potentially present and additionally facial
impediment and additionally low goal of gained picture stays a generally
inexplicable issue. As such, current calculations are yet to be created.

Face Reocognition Application is utilized to naturally distinguish an individual from a

picture or a video source. The Recognition task is performed by getting facial highlights from
a picture of the subject's face. The principle objective of video-based face acknowledgment is
to distinguish a video face-track of popular characters utilizing a huge word reference of still
face pictures, while dismissing obscure people Existing techniques utilize probabilistic
models on a casing by-outline premise to recognize faces which is computationally costly
when the information size is huge

4
 TABLE OF CONTENTS
Certificate …………………………………………………………...…….....2
Acknowledgement …………………………………………………………...3
Abstract ……………………………………………………………………....4

1.Introduction .................................................................................................6
2.Software requirement analysis ...................................................................7
2.1 Functional requirements .......................................................................7
2.2 Nonfunctional requirements .................................................................7
2.3 Defining the problem ............................................................................7

3. Software Design ...........................................................................................8

3.1 Fig: Architecture of video based face recognition ................................8

4. Software and Hardware requirements ......................................................9

4.1 Software/ libraries required .............................................................9
4.2 Hardware requirements ...................................................................9

5.Coding/Code Templates ..............................................................................10

6.Testing ...........................................................................................................29
7.Output ...........................................................................................................33
8.Conclusions ...................................................................................................35
9.Bibliography .................................................................................................35
 1. INTRODUCTION

While current face acknowledgment frameworks perform well under moderately controlled
conditions , they tend to endure when varieties in posture, brightening or looks are available.
Then again, the human visual framework has momentous capacities to perceive faces even
under more regrettable review conditions. Normally, the human insight not just uses the
facial construction to perceive faces yet additionally extra signals like tone, facial movement,
relevant information and so forth Critically, ongoing psychophysical examines show that
facial development upholds the face acknowledgment measure particularly in corrupted
conditions .

Enlivened by these discoveries, specialists and engineers have as of late endeavored to

abuse facial elements to upgrade still picture based face acknowledgment. Nonetheless, a
large portion of these endeavors don't misuse facial elements proficiently however apply
still-picture based strategies to some "great" outlines chose from face successions . a face
acknowledgment framework for video is accordingly proposed.

The framework depends on following the nose and eye positions. The areas of these
three focuses are utilized to choose whether the direction of the face is appropriate for
face acknowledgment

6
 2. Software Requirement Analysis

2.1 Functional Requirements

• Input: The application is designed to work on a video frame which it will

process and take facial features and compare with the known faces.
• Operations: Adding Data(Images), Processing Data(Image Processing
and encoding), Deleting the Data(Removing Stored Images), Real time
face recognition, Face Recognition in a recorded video.
• Outputs: Returns the Frame of the video, highlighting the recognized face.

2.2 Non-Functional Requirements

• The running time of the program may vary with the availability of operating system
• Encoding the Images takes time according to the resources available.

2.2 Defining the problem

❖ To identify whether the application is able to recognize the

person in the video using person’s image as a reference.

7
 3. Software Design

Fig 3.1 Working of Video based face recognition.First, the input is given in the form of
Video in which face has to be recognized. Then, face of the object is tracked in the video
present and after that, the tracked face frame is taken and Feature Extraction takes place.
These Features are then matched with the features of the given image of whom is has to
recognized and then compared both the images. After that, results are produced.

8
 4. Software and Hardware Requirements
4.1 Software / Libraries Requirement

• Python
• VisualCode
• OpenCV
• Numpy
• Pillow
• Tkinter
• Dlib
• Os
• Shutil
• Unittest

4.2 Hardware Requirements

• Processor: Intel® Core(TM) i5-5200 CPU @ 2.20 GHz

• RAM: 4 GB RAM
• 64-bit OS, x64-based processor

9
6. Coding/Code Templates

10
The Image and Buttons are placed in 2 different frames except the exit button is
not in any frame but on the root screen itself. Pillow is used to read the image file
and pack it in the frame to make the application more presentable.

Root.mainloop is used to keep the window active or the program end

instantly after execution.

Different Buttons are created using the button class and a proper command is
passed which is executed when the button is clicked and text field shows the text on
the button.

LabelFrames class is used to create Image Frame where image is placed and
button frame where the four main buttons are placed.

Grid and Pack functions are used to add the buttons and frames in the screen. Grid
is used when we have to give proper placement to a button or a frame whereas
Pack function just places the button or frame and packs it in the window screen.

11
Add Button

The Add Data button invoke a toplevel class which creates a window over our root
window. We used one frame, in which the image are browsed and name of the
person in the image is provided and on the screen “Save” and “Exit” Buttons are
placed to save the provided data or exit the Add data window.

12
This is the set of code used to develop the add data window

13
 The functionalities added to the save button is given below

In this we are using shutil to copy the image file using its path to our “known_faces”
folder from where our application us encoding the data for comparison.

14
Below is the browse button functionality which takes image_label that is right beside the
button as a parameter. Python Tkinter (and TK) offer a set of dialogs that you can use when
working with files. By using these you don’t have to design standard dialogs your self.
Example dialogs include an open file dialog, a save file dialog and many others. Besides
file dialogs there are other standard dialogs, but in this article we will focus on file dialogs.

After getting appropriate data the image is added in the “known_faces” folder.

15
 Right after this your provided data will be added to the “known_faces folder”.

Delete Button

This is the delete button window which takes input only the name of the person whose data
is to be deleted.

16
Here is the code used in development of the delete window. It also uses the toplevel class
which means this window is also placed with the root window.

The only label is the “name” label and “Entry” class is used for taking input from the user.
The delete button removes the data of the person whose name is inputed and exit button is
used to close the window.

Above you can see, that previously added data of Barack Obama is deleted.

Real Time

Real Time Button invokes the webcam using opencv and uses the api function “play”
which shows real time webcam feed with the “compare” function which detects and creates
a box around the identified persons face.

17
This is the play functionality which takes video as a parameter, In case of Real_time button
your webcam feed is provided. The play function reads the video frame by frame and use the
compare function to compare the person in the video to its known data.

Video Button

This is the functionality of the video button. It uses Tkinter’s file dialog to get the path of
video in which a person is to be recognized, then provides the path to videoCapture function
which reads the video and then use the same play function to play the video and identify the
faces present in it.

This is a video of Elon Musk in which our face recognition system is able to detect
elon musk’s face in it.

18
 cp.py

This is a part of backend for the GUI in which video processing function and compare
function is developed using OOPs. For this file we have used tkinter, os, opencv and
developed face_recognition file.

The class constructor initializes two list variables which contains the encodings of known
persons image and its name. The for loop, loops through all the file and append the
encoding data, and known persons name in the lists. All this is done in constructor, so that
when the application is executed it stores the data in the very beginning.

After initialization of know encodings and known persons name, we need to process the
video frame by frame. To do that we have used opencv which helps in processing the video.
Once the play function starts exeuting, each video frame is passed in to compare function as
parameter.

19
We have used try and except to handle any exception which may occur while processing the
video file. If any exception is occures during the processing of the video a message
dialogue will appear so that the application won’t crash abruptly.

20
Next we have the compare function which is used to compare known face with the face
detected in the video.

The compare function detects the face in the image and use its location to encode the face
landmarks, then we use the for loop to check for all the faces detected in the passed image
frame and compare it to the known encodings. Compare_faces function returns a list of
boolean and for all the True values in the list we are creating two rectangles, one around the
face and second below the first one which has the persons name. To achieve this we have
used Opencv. The first box which is created around the face is of green color. It is done by
passing the color coding in for of list “[0,255,0]” the three value signifies Red Green Blue
color code. As for the second box we have “[0,0,0]” this list and it is black in color as you
can see in the earlier image.

21
 Face_recognition file
This file acts as the main interface for face detection and recognition part of the whole
workflow. The functions which initiate and perform the processes such as face detection,
shape prediction, landmarking, face recognition, encoding, object conversion etc. are defined
in this file.

In the above snippet all the modules that we need are imported and dlib models which are
necessary are taken. Since the lib

Get_frontal_face_detector : The get_frontal_face_detector will

return a detector that is a function we can use to retrieve the faces information.
Each face is an object that contains the points where the image can be found.

Shape_predictor_68_face_landmarks: dlib.get_frontal_face_detector () is used in detecting

the face in a frame or image.dlib.shape_predictor() is a tool that takes in an image region
containing some object and outputs a set of point locations that define the pose of the object

Dlib_face_recognition_resnet_model_v1: Dlib offers a high-quality face recognition

algorithm based on deep learning. Dlib implements a face recognition algorithm that offers
state-of-the-art accuracy. More specifically, the model has an accuracy of 99.38% on the
labeled faces in the wild database.

22
The implementation of this algorithm is based on the ResNet-34 network proposed in the
paper Deep Residual Learning for Image Recognition (2016), which was trained using
three million faces.

Rect2css: Converts a dlib rect (rectangle for short) object to tuple form in (top, right,
bottom left). Takes parameter a dlib 'rect' object a plain tuple representation of the rect in
(top, right, bottom, left) order

Css2rect: Convert a tuple in (top, right, bottom, left) order to a dlib `rect` object. Takes
parameter plain tuple representation of the rect in (top, right, bottom, left) order and returns
a dlib `rect` object

Conversion is needed as some function need object in rect form and some need in css
form.

Inbound(): Makes sure a tuple in (top, right, bottom, left) order is within the bounds of the
image. Returns a trimmed plain tuple representation of the rect in (top, right, bottom, left)
order. Main function is to make sure the css object remains inbounds (in boundaries) of the
needed image.

23
Load_image_file (): As the name suggests it loads all the images files in different forms to a
numpy array for faster processing since lists in python are slow.

Raw_face_locations (): Returns an array of bounding boxes of human faces in a image.

Takes parameter an image (as a numpy array). This function locates the raw facial area
and returns the detected part of the face. Uses the dlib detector model.

Face_locations (): Returns an array of bounding boxes of human faces in a image. Takes
parameters as An image (as a numpy array). Marks the face locations detected by previous
function.

24
Example:

Face locations are marked in green rectangle objects.

Raw_face_landmarks (): When provided an image, returns landmarks aka points of face
feature locations (eyes, nose, etc.) for each face in the image. As the name of model suggests
there are 68 such landmarks. Returns a list of landmarks identified in the face locations taken
input from face_locations function.

25
Example of landmarks:

Facial features such as eyes, nose etc. are taken in point forms known as landmarks.

26
Face_encodings (): Given an image, return the 128-dimension face encoding for each face
in the image. Takes list as input sent from raw landmarks function and encodes the numpy
array and pass it in list form to compare faces function.

Example: Encoded image in form of array

27
Compare_faces (): Compare a list of face encodings against a candidate
encoding to see if they match. return: A list of True/False values indicating which
known_face_encodings match the face encoding to check.

Difference between known_face_encodings and

unknown_face_encodings and normalize it using linalg function of numpy and is
compared to tolerance. By default tolerance is 0.6 and if difference obtained is equal or
less than tolerance the known face is detected.

This is how face detection and recognition is carried out using face recognition file in
this project.

28
 6. TESTING

This part of the code is to test the face_recognition.py file. In this we have used different
images to test the functions of face_recognition file and those functions are load_image_file,
raw_face_landmarks, face_locations, raw_face_locations, face_encodings and compare.
These functions returns list a values of which contains the facial features. To develop this
file we have used unittest library of python which is an inbuilt library.

load_image_file function is responsible for loading image file using Python imaging library
or Pillow. To test it we are comparing the image shape by loading it. We have three test
cases which are images Barack Obama, Joe Biden and an Image containing photos of Joe
Biden, Elon musk, Barack Obama and Kim Kardashian.

The code to test the loaded image file.

In the above image you can see that load_image_file function is used to load the image
file and assertEqual check the image shape with the original shape.

29
Next we are testing the raw_face_locations function. In raw_face_locations we are
checking for the no. Of faces that are detected in the image file

Now we will test face_locations functions which gives the locations of faces.

30
test_raw_face_landmarks function is used to mark the face_landmarks on the detected face
and returns the location of all the 68 point landmarks. To test this file we have compared
the number of face landmarks detected on the face.

face_encoding function is used to encode the detected face and return the encoded list of
the image. In this file we are testing for the number of faces detected and the number of
encodings.

31
Test_compare_faces function is used to compare two faces. To test this function we have

compared the Barack Obama’s picture with another image of his for the first test case. In
the other test case we have compared Barack Obama’s picture with Joe Biden. To test this
we have used assertTrue and assertFalse

32
8. OUTPUT

33
34
 8. CONCLUSION
The Video Based Face Recognition application can be widely use in security areas. The
Application has two phases which are feature extraction and comparing the extracted
features of input image and the frame of the video. In the first phase, the feature extraction
technique is applied and in the second phase, The comparison between the encodings of the
two images is done and according to that the result is obtain.
This can be used to detect whether a someone who escape from the prison is in the frame or
not or who have done the robbery. It can also be used to recognize any suspended student
who is not allowed in college premises etc.

9.BIBLIOGRAPHY
1. Video-Based Face Recognition Using Probabilistic Appearance Manifolds by
Kuang-Chih Lee Jeffrey Ho, Ming-Hsuan Yang, David Kriegman, klee10.
2. https://www.pyimagesearch.com/2021/04/19/face-detection-with-dlib-hog-and-
cnn/
3. https://www.pyimagesearch.com/2018/06/18/face-recognition-with-opencv-
python-and-deep-learning/
4. https://docs.python.org/3/library/tkinter.html
5. https://docs.python.org/3/library/unittest.html

35