“Extracting Face from Live Feed”

A DISSERTATION
Submitted in partial fulfillment of the requirement for the award of the degree of Bachelor of
Engineering

[ELECTRONICS & COMMUNICATION ENGINEERING]

Submitted To

RAJIV GANDHI PRODYOGIKI VISHWAVIDYALAYA,

(University of Technology of Madhya Pradesh) BHOPAL (M.P.)

[ELECTRONICS & COMMUNICATION ENGINEERING]

Submitted by: Under the Supervision of:

SHANU PRAKASH [0133EC161141] Dr. Sangeeta Shukla

SACHIN VISHWS [0133EC161121] ECE Department

SATYAJEET KUMAR [0133EC161131]

Department of Electronics & Communication Engineering

SAGAR INSTITUTE OF RESEARCH & TECHNOLOGY, Bhopal

[2019-2020]
 Sagar Institute of Research & Technology, Bhopal
Department of Electronics & Communication Engineering

CERTIFICATE
This is to certify that the work embodies in this dissertation entitled“ Extracting

Face from Live Feed ”being Submitted by Shanu Prakash Enrolment no.-
0133EC161141 in partial fulfillment of the requirement for the award of
“Bachelor of Engineering in Electronics & Communication Engineering” to RAJIV
GANDHI PRODYOGIKI VISHWAVIDYALAYA,BHOPAL(M.P.) during the academic
year 2019-2020is a record of Bonafede piece of work, carried out by him under
my supervision and guidance in the Department of Electronics & Communication
Engineering ,Sagar Institute of Research & Technology, Bhopal

Bhopal.

GUIDED BY: DR. SANGEETA SHUKLA

PROJECT-COORDINATOR: PROFFESOR. MUKESH YADAV

HOD ECE DEPARTMENT: DR. JYOTI JAIN

 Sagar Institute of Research & Technology, Bhopal
Department of Electronics & Communication Engineering

DECLARATION

I Shanu Prakash a student of “Bachelor of Engineering in Electronics &

Communication Engineering” ,session – 2019-2020, here by informed that the

work presented in this dissertation entitled “Extracting Faces from Live Feed” is

the outcome of my own work, is Bonafede and correct to the best of my

knowledge and this work has been carried out taking care of Engineering Ethics.

The work presented does not infringe any patented work and has not been

submitted to any other University or anywhere else for the award of any degree

or any professional diploma.

NAME – Shanu Prakash

Enrollment No. – 0133EC161141

 ACKNOWLEDGEMENT

I take an opportunity to acknowledge and extend my heartfelt gratitude to my

guide Dr. Sangeeta Shukla for all advice, encouragement and constant support.
She has given us throughout my project work. This work would not have been
possible without her support and valuable suggestions.

I would like to sincerely thanks Dr. Jyoti Jain, HOD. Electronics & Communication
Department for her great support in doing this project.

I am also grateful to Prof. Mukesh Yadav for giving me the necessary facilities to
carry out my project work successfully.

I would like to thanks all my friends for their help and constructive criticism during
my project work. Finally, I have no words to express my sincere gratitude to my
parents who have been shown me this world and for every support they have
given us

Shanu Prakash

0133EC161141
 CONTENTS

S. No Topic Page No.

01 Abstract
02 Introduction
03 Face Detection & Algorithm
04 Facial Recognition
05 Objective
06 Python
07 Computer Vision
08 OpenCV
09 Project Structure
10 Project
11 Conclusion
12 References
 ABSTRACT

A face recognition system is one of the biometric information processes, its

applicability is easier and working range is wider than other systems like;
fingerprint, iris scanning, signature, etc. The detection methods are designed
to extract features of face region out of a digital image. The output face image
of the detection algorithm should be similar to the recognition input image.
Face detection is performed on live acquired images without any application
field in mind. The developed system uses white balance correction, skin like
region segmentation, facial feature extraction and face image extraction on a
face candidate. System is also capable of detecting multiple faces in live
acquired images.
 INTRODUCTION

Face recognition systems are part of facial image processing applications and
it is one of the challenging topics in IT research. Biometric information of
humans are used instead of fingerprint, iris, signature etc., because these
types of biometrics are not much suitable for non-collaborative people. Face
recognition systems are usually used and preferred for security cameras in
metropolitan life. These systems can be used for crime prevention, video
surveillance, person verification, and similar security activities.

Face detection is the first step of face recognition system. Output of the
detection is location of a face region with facial features (i.e. eyes, mouth,
eyebrow, nose etc.) [1]. Detection methods in the literature are classified into
two groups as Knowledge-Based Methods and Image-Based Methods.

Facial recognition system are commonly used for security purposes but are
increasingly being used in a variety of other applications. For example

 Facebook uses facial recognition software to help automate user tagging in

photographs.
 Mobile Phones use Facial Recognition System as a password.
 It is also used in industries as attendance system.
What are biometrics?
A biometric is a unique, measurable characteristic of a human being that can be
used to automatically recognize an individual or verify an individual identity.
Biometrics can measure both physiological and behavioral characteristics.

Physiological Biometrics:-

Based on measurements and data derived from direct the human body

 Finger Print Scan

 Facial Recognition
 Iris Scan
 Retina Scan
 Hand scan

Behavioral Biometrics:-

Based on measurements and data derived from an action

 1.Voice scan
 2.Signature scan

Knowledge-Based methods use information about Facial Features, Skin

Color or Template Matching. Facial Features are used to find eyes, mouth,
nose or other facial features to detect the human faces. Skin color is one of
the most significant features of human face. Skin color can be modeled
with parameterized or non-parameterized methods. Skin color region can
be identified in terms of threshold region, elliptical modeling, statistical
modeling (i.e. Gaussian Modeling), or Neural Network. Skin color is
modeled in each color spaces like RGB, YCbCr, HSV, YUV, and in statistical
models. Face has a unique pattern to differentiate from other objects and
hence a template can be generated to scan and detect faces.

Image-Based methods use training/learning methods to make comparison

between face and non- face images. For these methods, large number of
images of face and non-face should be trained to increase the accuracy of
the system. AdaBoost, Eigenface, Neural Networks and Support Vector
Machines are kind of methods that are used commonly in face detection
algorithms.

There are many more different ways to perform this operation

TensorFlow is a platform which can be used to train your own model and
the trained mobel can be used to detect any kind of data such as Pen, TV,
Computer, Laptop, Bottle etc. depending upon the model made by you

Many of the face or Object Detection models use Python Environment to run.
 FACE DETECTION ALGORITHM

Face detection performs locating and extracting face image operations for
face recognition system. Face detection algorithm of the proposed system.

Our experiments reveal that skin segmentation, as a first step for face
detection, reduces computational time for searching whole image. While
segmentation is applied, only segmented region is searched whether the
segment includes any face or not. After the acquired image is segmented, it
continued with the face candidate search and facial feature extraction to
validate the candidate. To perform the facial feature extraction four different
filtering approaches are performed for useful enhancements on facial
features. Finally, the validated face is extracted based on the facial features
to send to the face recognition algorithm.

 Knowledge-Based
 Feature Based
 Template Match
 FACIAL RECOGNITION

The face is an important part of who you are and how people identify you.

For face recognition there are two types of comparison.

The first is verification and second is identification.

Verification is where the system compares the given individual with who that
individual says they are and gives a yes or no decision.

Identification is where the system compares the given individual to all the other
individuals in the database and gives a ranked list of matches.

All identification or authentication technologies operate using the following four

stages:

1. Capture: A physical sample is captured by the system during enrollment

and also in identification or verification process.
2. Extraction: Unique data is extracted from the sample and a template is
created.
3. Comparison: The template is then compared with a new sample.
4. Match/Non-match: The system decides if the features extracted from the
new one.
 OBJECTIVE

The project has two essential elements:

1. Box Around Faces: Show white boxes around all the faces recognized in
the image.
The Python file is data_generator.py

2. Extracted Faces: Extract faces from all images in a folder and save each
face into a destination folder to create a handy dataset.
The Python file is face_extractor.py

Software used in the Project:

 Python (Anaconda Spyder IDE)

Module Used:
 Computer Vision (OpenCV)
 Time Module
 NumPy
 OS Module
 PYTHON

Python is a general purpose programming language started by Guido van

Rossum, which became very popular in short time mainly because of its simplicity
and code readability. It enables the programmer to express his ideas in fewer
lines of code without reducing any readability.

Compared to other languages like C/C++, Python is slower. But another important
feature of Python is that it can be easily extended with C/C++. This feature helps
us to write computationally intensive codes in C/C++ and create a Python wrapper
for it so that we can use these wrappers as Python modules.

This gives us two advantages:

 Our code is as fast as original C/C++ code (since it is the actual

C++ code working in background)
 It is very easy to code in Python.

Python is considered a scripting language, like Ruby or Perl and is often used for
creating Web applications and dynamic Web content. It is also supported by a
number of 2D and 3D imaging programs, enabling users to create custom plug-
ins and extensions with Python. Examples of applications that support a
Python API include GIMP, Inkscape, Blender, and Autodesk Maya.

Scripts written in Python (.PY files) can be parsed and run immediately. They can
also be saved as a compiled programs (.PYC files), which are often used as
programming modules that can be referenced by other Python programs.
The Python 2 language, i.e. Python 2.7.x, was officially discontinued on 1 January
2020 (first planned for 2015) after which security patches and other
improvements will not be released for it. With Python 2's end-of-life, only
Python 3.5.x and later are supported.

Python interpreters are available for many operating systems. A global

community of programmers develops and maintains CPython, an open
source reference implementation. A non-profit organization, the Python Software
Foundation, manages and directs resources for Python and CPython
development.

There are many different Python Programming IDE available for different
Operating Systems.

Some of them are:

 Python IDLE
 PyCharm
 Anaconda
 Spyder IDE
 Jupyter NoteBook

Every Python Development Software uses a Python Environment to perform.

Python IDLE:

IDLE is Python’s Integrated Development and Learning Environment.

IDLE is an integrated development environment for Python, which has been

bundled with the default implementation of the language since 1.5.2b1. It is
 packaged as an optional part of the Python packaging with many Linux
distributions. It is completely written in Python and the Tkinter GUI toolkit.

IDLE has the following features:

 Coded in 100% pure Pyth on, using the tkinter GUI toolkit
 Cross-platform: works mostly the same on Windows, Unix, and
macOS
 Python shell window (interactive interpreter) with colorizing of
code input, output, and error messages
 Multi-window text editor with multiple undo, Python
colorizing, smart indent, call tips, auto completion, and other
features
 Search within any window, replace within editor windows, and
search through multiple files (grep)
 Debugger with persistent breakpoints, stepping, and viewing
of global and local namespaces
 Configuration, browsers, and other dialogs
PyCharm:-

PyCharm is an integrated development environment (IDE) used

in computer programming, specifically for the Python language. It is
developed by the Czech company JetBrains. It provides code analysis,
a graphical debugger, an integrated unit tester, integration
with version control systems (VCSes), and supports web
development with Django as well as Data Science with Anaconda.

PyCharm is cross-platform, with Windows, macOS and Linux versions.

The Community Edition is released under the Apache License, and
there is also Professional Edition with extra features.

Features
 Coding assistance and analysis, with code completion, syntax
and error highlighting, linter integration, and quick fixes
 Project and code navigation: specialized project views, file
structure views and quick jumping between files, classes,
methods and usages
  Python refactoring: includes rename, extract method,
introduce variable, introduce constant, pull up, push down and
others
 Integrated Python debugger
 Integrated unit testing, with line-by-line code coverage
 Google App Engine Python development [professional edition
only]
 Version control integration: unified user interface
for Mercurial, Git, Subversion, Perforce and CVS with change
lists and merge
 Support for scientific tools like matplotlib, numpy and scipy
[professional edition only]

PyCharm is very demanding now a day due the features offered by

the Software
 Fig: PyCharm Start Screen

Anaconda:-

The open-source Anaconda Individual Edition (formally Anaconda

Distribution) is the easiest way to perform Python/R data science and
machine learning on Linux, Windows, and Mac OS X. With over 19
million users worldwide, it is the industry standard for developing,
testing, and training on a single machine, enabling individual data
scientists to:
 Quickly download 7,500+ Python/R data science packages
 Manage libraries, dependencies, and environments with Conda
 Develop and train machine learning and deep learning models
with scikit-learn, TensorFlow, and Theano
 Analyze data with scalability and performance
with Dask, NumPy, pandas, and Numba
 Visualize results with Matplotlib, Bokeh, Datashader,
and Holoviews
Anaconda is a scientific Python distribution. It has no IDE of its own. The default
IDE bundled with Anaconda is Spyder which is just another Python package that
can be installed even without Anaconda.

Anaconda bundles a whole bunch of Python packages that are commonly used by
people using Python for scientific computing and/or data science. It provides a
single download and an install program/script that installs all the packages in one
go. Alternatively, one can install Python first and then individually install all the
required packages using pip. Additionally, it provides its own package manager
(conda) and package repository. But it allows installation of packages from PyPI
using pip if the package is not in Anaconda repositories. It is especially good if you
are installing on Microsoft Windows as it can easily install packages that would
otherwise require you to install C/C++ compilers and libraries if you were using
pip. It is certainly an added advantage that conda, in addition to being a package
manager, is also a virtual environment manager allowing you to install
independent development environments and switch from one to the other
(similar to virtualenv).

Fig: Anaconda Navigation Panel

Spyder IDE Software:

Spyder is an open source cross-platform integrated development

environment (IDE) for scientific programming in the Python language. Spyder
integrates with a number of prominent packages in the scientific Python stack,
including NumPy, SciPy, Matplotlib, pandas, IPython, SymPy and Cython, as well
as other open source software. It is released under the MIT license.

Initially created and developed by Pierre Raybaut in 2009, since 2012 Spyder has
been maintained and continuously improved by a team of scientific Python
developers and the community.

Spyder is extensible with first- and third-party plugins, includes support for
interactive tools for data inspection and embeds Python-specific code quality
assurance and introspection instruments, such as Pyflakes, Pylint and Rope. It is
available cross-platform through Anaconda, on Windows, on macOS
through MacPorts, and on major Linux distributions such as Arch Linux, Debian,
Fedora, Gentoo Linux, openSUSE and Ubuntu.

Spyder uses Qt for its GUI, and is designed to use either of

the PyQt or PySide Python bindings. QtPy, a thin abstraction layer developed by
the Spyder project and later adopted by multiple other packages, provides the
flexibility to use either backend.

Features

Features include:

 An editor with syntax highlighting, introspection, code completion

 Support for multiple IPython consoles
 The ability to explore and edit variables from a GUI
  A Help pane able to retrieve and render rich text documentation on
functions, classes and methods automatically or on-demand
 A debugger linked to IPdb, for step-by-step execution
 Static code analysis, powered by Pylint
 A run-time Profiler, to benchmark code
 Project support, allowing work on multiple development efforts
simultaneously
 A built-in file explorer, for interacting with the filesystem and managing
projects
 A "Find in Files" feature, allowing full regular expression search over a
specified scope
 An online help browser, allowing users to search and view Python and
package documentation inside the IDE
 A history log, recording every user command entered in each console
 An internal console, allowing for introspection and control over Spyder's
own operation

Plugins

Available plugins include:[13]

 Spyder-Unittest, which integrates the popular unit testing frameworks

Pytest, Unittest and Nose with Spyder
 Spyder-Notebook, allowing the viewing and editing of Jupyter
Notebooks within the IDE
 Spyder-Reports, enabling use of literate programming techniques in Python
 Spyder-Terminal, adding the ability to open, control and manage cross-
platform system shells within Spyder
 Spyder-Vim, containing commands and shortcuts emulating the Vim text
editor
 Spyder-AutoPEP8, which can automatically conform code to the standard
PEP 8 code style
  Spyder-Line-Profiler and Spyder-Memory-Profiler, extending the built-in
profiling functionality to include testing an individual line, and
measuring memory usage

It is highly demanding IDE for Data Science and Machine Learning.

Fig: Spyder IDE

Jupyter NoteBook:-

Jupyter Notebook (formerly IPython Notebooks) is a web-based

interactive computational environment for creating Jupyter notebook documents.
The "notebook" term can colloquially make reference to many different entities,
mainly the Jupyter web application, Jupyter Python web server, or Jupyter
document format depending on context. A Jupyter Notebook document is
a JSON document, following a versioned schema, and containing an ordered list of
input/output cells which can contain code, text (using Markdown), mathematics,
plots and rich media, usually ending with the ".ipynb" extension.

The Jupyter Notebook is an open-source web application that allows you to

create and share documents that contain live code, equations, visualizations and
narrative text. Uses include: data cleaning and transformation, numerical
simulation, statistical modeling, data visualization, machine learning, and much
more

Jupyter Notebook provides a browser-based REPL built upon a number of

popular open-source libraries:

 IPython
 Tornado (web server)
 jQuery
 Bootstrap (front-end framework)
 MathJax

Fig: Jupyter Notebook

Jupyter Notebook can be accessed using any web-browser such as:

 Google Chrome
 FireFox
 Microsoft Edge
 Safari Web Browser (MacOS)
 Computer Vision

Computer vision is an interdisciplinary scientific field that deals with

how computers can gain high-level understanding from digital images or videos.
From the perspective of engineering, it seeks to understand and automate tasks
that the human visual system can do.

Computer vision tasks include methods for acquiring, processing, analyzing and
understanding digital images, and extraction of high-dimensional data from the
real world in order to produce numerical or symbolic information, e.g. in the
forms of decisions. Understanding in this context means the transformation of
visual images (the input of the retina) into descriptions of the world that make
sense to thought processes and can elicit appropriate action. This image
understanding can be seen as the disentangling of symbolic information from
image data using models constructed with the aid of geometry, physics, statistics,
and learning theory.

The scientific discipline of computer vision is concerned with the theory behind
artificial systems that extract information from images. The image data can take
many forms, such as video sequences, views from multiple cameras, multi-
dimensional data from a 3D scanner or medical scanning device. The
technological discipline of computer vision seeks to apply its theories and models
to the construction of computer vision systems.

Sub-domains of computer vision include scene reconstruction, event

detection, video tracking, object recognition, 3D pose estimation, learning,
indexing, motion estimation, visual servoing, 3D scene modeling, and image
restoration.
Definition

Computer vision is an interdisciplinary field that deals with how computers can be
made to gain high-level understanding from digital images or videos. From the
perspective of engineering, it seeks to automate tasks that the human visual
system can do. "Computer vision is concerned with the automatic extraction,
analysis and understanding of useful information from a single image or a
sequence of images. It involves the development of a theoretical and algorithmic
basis to achieve automatic visual understanding." As a scientific discipline,
computer vision is concerned with the theory behind artificial systems that
extract information from images. The image data can take many forms, such as
video sequences, views from multiple cameras, or multi-dimensional data from
a medical scanner. As a technological discipline, computer vision seeks to apply its
theories and models for the construction of computer vision systems.

Fields where Computer Vision is Used:

Artificial intelligence

Areas of artificial intelligence deal with autonomous planning or deliberation for

robotic systems to navigate through an environment. A detailed understanding of
these environments is required to navigate through them. Information about the
environment could be provided by a computer vision system, acting as a vision
sensor and providing high-level information about the environment and the
robot.

Artificial intelligence and computer vision share other topics such as pattern
recognition and learning techniques. Consequently, computer vision is sometimes
seen as a part of the artificial intelligence field or the computer science field in
general.

Information Engineering

Computer vision is often considered to be part of information engineering

Solid-state physics

Solid-state physics is another field that is closely related to computer vision. Most
computer vision systems rely on image sensors, which detect electromagnetic
radiation, which is typically in the form of either visible or infra-red light. The
sensors are designed using quantum physics. The process by which light interacts
with surfaces is explained using physics. Physics explains the behavior
of optics which are a core part of most imaging systems. Sophisticated image
sensors even require quantum mechanics to provide a complete understanding of
the image formation process.[11] Also, various measurement problems in physics
can be addressed using computer vision, for example motion in fluids.

Neurobiology

A third field which plays an important role is neurobiology, specifically the study
of the biological vision system. Over the last century, there has been an extensive
study of eyes, neurons, and the brain structures devoted to processing of visual
stimuli in both humans and various animals. This has led to a coarse, yet
complicated, description of how "real" vision systems operate in order to solve
certain vision-related tasks. These results have led to a subfield within computer
vision where artificial systems are designed to mimic the processing and behavior
of biological systems, at different levels of complexity. Also, some of the learning-
based methods developed within computer vision (e.g. neural net and deep
learning based image and feature analysis and classification) have their
background in biology.

Some strands of computer vision research are closely related to the study
of biological vision – indeed, just as many strands of AI research are closely tied
with research into human consciousness, and the use of stored knowledge to
interpret, integrate and utilize visual information. The field of biological vision
studies and models the physiological processes behind visual perception in
humans and other animals. Computer vision, on the other hand, studies and
describes the processes implemented in software and hardware behind artificial
vision systems. Interdisciplinary exchange between biological and computer vision
has proven fruitful for both fields.[19]

Signal Processing

Yet another field related to computer vision is signal processing. Many methods
for processing of one-variable signals, typically temporal signals, can be extended
in a natural way to processing of two-variable signals or multi-variable signals in
computer vision. However, because of the specific nature of images there are
many methods developed within computer vision which have no counterpart in
processing of one-variable signals. Together with the multi-dimensionality of the
signal, this defines a subfield in signal processing as a part of computer vision.
 OpenCV

OpenCV was started at Intel in 1999 by Gary Bradsky and the first release came
out in 2000. Vadim Pisarevsky joined Gary Bradsky to manage Intel’s Russian
software OpenCV team. In 2005, OpenCV was used on Stanley, the vehicle who
won 2005 DARPA Grand Challenge. Later its active development continued under
the support of Willow Garage, with Gary Bradsky and Vadim Pisarevsky leading the
project. Right now, OpenCV supports a lot of algorithms related to Computer
Vision and Machine Learning and it is expanding day-by-day.

Currently OpenCV supports a wide variety of programming languages like C++,

Python, Java etc and is available on different platforms including Windows, Linux,
OS X, Android, iOS etc. Also, interfaces based on CUDA and OpenCL are also under
active development for high-speed GPU operations.

OpenCV-Python is the Python API of OpenCV. It combines the best qualities of

OpenCV C++ API and Python language.

OpenCV (Open Source Computer Vision Library: http://opencv.org) is an open-

source BSD-licensed library that includes several hundreds of computer vision
algorithms. The document describes the so-called OpenCV 2.x API, which is
essentially a C++ API, as opposed to the C-based OpenCV 1.x API (C API is
deprecated and not tested with "C" compiler since OpenCV 2.4 releases)

OpenCV has a modular structure, which means that the package includes several
shared or static libraries. The following modules are available:

 Core functionality (core) - A compact module defining basic data

structures, including the dense multi-dimensional array Mat and basic
functions used by all other modules.
 Image Processing (imgproc) -An image processing module that includes
linear and non-linear image filtering, geometrical image transformations
 (resize, affine and perspective warping, generic table-based remapping),
color space conversion, histograms, and so on.
 Video Analysis (video) - A video analysis module that includes motion
estimation, background subtraction, and object tracking algorithms.
 Camera Calibration and 3D Reconstruction (calib3d) - Basic multiple-view
geometry algorithms, single and stereo camera calibration, object pose
estimation, stereo correspondence algorithms, and elements of 3D
reconstruction.
 2D Features Framework (features2d) - Salient feature detectors,
descriptors, and descriptor matchers.
 Object Detection (objdetect) - Detection of objects and instances of the
predefined classes (for example, faces, eyes, mugs, people, cars, and so on).
 High-level GUI (highgui) - An easy-to-use interface to simple UI capabilities.
 Video I/O (videoio) - An easy-to-use interface to video capturing and video
codecs.

OpenCV's application areas include:

 2D and 3D feature toolkits

 Egomotion estimation
 Facial recognition system
 Gesture recognition
  Human–computer interaction (HCI)
 Mobile robotics
 Motion understanding
 Object identification
 Segmentation and recognition
 Stereopsis stereo vision: depth perception from 2 cameras
 Structure from motion (SFM)
 Motion tracking
 Augmented reality

How OpenCV’s face recognition works

In order to build our OpenCV face recognition pipeline, we’ll be applying deep
learning in two key steps:

 To apply face detection, which detects the presence and location of a face in an
image, but does not identify it
 To extract the 128-d feature vectors (called “embeddings”) that quantify each
face in an image.

The model responsible for actually quantifying each face in an image is from
the Open Face project, a Python and Torch implementation of face recognition
with deep learning.

First, we input an image or video frame to our face recognition pipeline. Given the
input image, we apply face detection to detect the location of a face in the image.

Optionally we can compute facial landmarks, enabling us to pre-process.

Face alignment, as the name suggests, is the process of

(1) Identifying the geometric structure of the faces

(2) Attempting to obtain a canonical alignment of the face based on translation,
rotation, and scale.

While optional, face alignment has been demonstrated to increase face

recognition accuracy in some pipelines.

After we’ve (optionally) applied face alignment and cropping, we pass the input
face through our deep neural network:

Figure 1: An overview of the OpenCV face recognition pipeline. The key step is a CNN feature
extractor that generates 128-d facial embeddings.
The model responsible for actually quantifying each face in an image is from
the Open Face project, a Python and Torch implementation of face recognition
with deep learning.

First, we input an image or video frame to our face recognition pipeline. Given the
input image, we apply face detection to detect the location of a face in the image.

Face alignment, as the name suggests, is the process of

 Identifying the geometric structure of the faces

 Attempting to obtain a canonical alignment of the face based on
translation, rotation, and scale.

While optional, face alignment has been demonstrated to increase face

recognition accuracy in some pipelines.

After we’ve (optionally) applied face alignment and cropping, we pass the input
face through our deep neural network:

The Face Net deep learning model computes a 128-d embedding that quantifies
the face itself.

But how does the network actually compute the face embedding?

The answer lies in the training process itself, including:

 The input data to the network

 The triplet loss function

To train a face recognition model with deep learning, each input batch of data
includes three images:

 The anchor
 The positive image
 The negative image
Figure 2: How the deep learning face recognition model computes the face embedding.
The anchor is our current face and has identity A.

The second image is our positive image — this image also contains a face of
person A.

The negative image, on the other hand, does not have the same identity, and
could belong to person B, C, or even Y!

The point is that the anchor and positive image both belong to the same
person/face while the negative image does not contain the same face.

The neural network computes the 128-d embeddings for each face and then
tweaks the weights of the network (via the triplet loss function) such that:

 The 128-d embeddings of the anchor and positive image lie closer together
 While at the same time, pushing the embeddings for the negative image
father away

In this manner, the network is able to learn to quantify faces and return highly
robust and discriminating embeddings suitable for face recognition.

And furthermore, we can actually reuse the Open Face model for our own
applications without having to explicitly train it!

Even though the deep learning model we’re using today has (very
likely) never seen the faces we’re about to pass through it, the model will still be
able to compute embeddings for each face — ideally, these face embeddings will
be sufficiently different such that we can train a “standard” machine learning
classifier (SVM, SGD classifier, Random Forest, etc.) on top of the face
embeddings, and therefore obtain our OpenCV face recognition pipeline.
.

Figure 3: A small example face dataset for face recognition with OpenCV
White Balance Correction

White balance of images differs due to change in lighting conditions of the

environment while acquiring image. This situation creates non-skin objects
that belong to skin objects. Therefore, white balance of the acquired image
should be corrected before segmenting it.

The implemented white balance algorithm is given below [34]:

 Calculate average value of red channel (Rav), green channel (Gav), and
blue channel (Bav)
 Calculate average gray, Grayav=(Rav+Gav+Bav)/3
 Then, KR=Grayav/Rav, KG=Grayav/Gav, and KB=Grayav/Bav
 Generate new image (New) from original image (Orji) with RGB
channel by New(R)=KR*Orj(R), New(G)=KG*Orj(G), and
New(B)=KB*Orj(B)

r”, “g”, and “b” parameters are red, green and blue channel values of pixel. If
these seven conditions are satisfied, then pixel is said to be skin color and
binary image is created from satisfied pixels.
Face Candidate Search

If the image is not balanced, then some part of the wall will be taken as skin
color as in Figure 3 due to poor lighting.

Under some lighting conditions, acquired image color can be wrong.

Therefore, this process will generate unwanted skin color like regions. To get
rid of this problem, segmentation is performed on both original image and
white balance corrected image. To create final skin image, logical “and
operation” is applied on both segmented original image and corrected image.
This operation will eliminate change of color value due to change of lighting
condition.

After “and operation” is applied on segmented images, noisy like small

regions are eliminated and morphological closing operation is applied to
merge gaps. Then, a region is said to be a face candidate if the ratio of
bounding box, width over height, lies between 0.3 and 1.5 and covering some
gaps inside the region.

Based on these conditions, face candidates are extracted from the input
image with modified bounding box. The height of bounding box modified as
1.28 times bigger than width of bounding box because width of face
candidate does not change if candidate includes chest/neck or not. After this
modification, new bounding box covers only face. These face candidates will
be sent to facial feature extraction part to validate the candidates.
Filtering for Facial Feature Extraction

Facial feature extraction bridges between detection and recognition parts.

First trial on extraction is made with profile extraction of face candidate.
Vertical profile of candidate is performed by taking mean of each row of
image matrix. Then, local minimum shows possible positions of eyebrow, eye,
nose tip, mouth, and chin. After eye position is determined in vertical profile,
then horizontal profile is extracted to determine eyes positions. Vertical and
horizontal profiles of two test face images are given in Figure 4 and 5.
Determination of exact position of eye position and mouth position is difficult
to determine in vertical profile. Also, it is difficult to determine position of
eyes in horizontal profile even the vertical position of eyes are determined in
vertical profile.

Due to difficulty in determination of position in vertical and horizontal

profiles in face candidate, face profile extraction is discarded and converting
Black- White image to find facial feature is performed. On the other hand,
binary thresholding is sensitive to lighting. If shadow appears on some part of
the face, some facial feature components can be eliminated. Some
experiments are performed and results are given in Figure 6.

Edge detection methods are nearly insensitive to light change. Sobel edge
detector is used to extract features. Figure 6 shows results of edge detection
on test image 3. Edge detection is not sensitive to light condition as Black-
White conversion. On both images, eyes and mouth can be selected with
human eyes but mouth can be difficult to extract on the images and eye parts
also vary on shapes. Also, edge detection has high responses.

In order to use edge detection, Laplacian of Gaussian (LoG) filter can be used.
LoG filter has low responses than edge detection. It makes useful
enhancements on facial features extraction. Figure 6 shows results of LoG
filter on test image 3.
 LoG filter has better performance than the previous three trials. Mouth is
more significant than others and eyes can be selected more accurately.

Facial Feature Extraction and Face Image

Facial feature is one of the most significant features of face. Facial features
are eyebrows, eyes, mouth, nose, nose tip, cheek, etc. If some of these
features are found in the candidate image, then the candidate image will be
considered as face. Two eyes and mouth generate isosceles triangle, and
distances between eye to eye and mid-point of eyes distance to mouth are
equal [6]. Some filtering operations are applied to extract feature candidates
and steps are listed below:

 Laplacian of Gaussian Filter on Red channel of candidate

 Contrast correction to improve visibility of filter result
 Average filtering to eliminate small noises
 Converting to binary image
 Removing small noises from binary images

After obtaining filtered image, labeling operation is applied to determine

which labels are possible to be facial features. Then, filtered image is divided
into three regions which is illustrated.

Right eye and left eye are determined by criteria which are illustrated by
yellow and green color respectively. Left and right eye are mostly found
correctly but sometimes bottom eyelid is found falsely. If left and right eyes
are detected, then mouth finding application can be applied.
 Each label inside down region chooses as mouth candidate and candidate
property vector is calculated. Euclidian distance of right eye to mouth
candidate (right-distance) and Euclidian distance of left eye to mouth
candidate (left-distance) are calculated. Also, Euclidian distance between two
eyes (eye-distance) and Euclidian distance between mid-point of eyes to
mouth candidate (center-distance) are calculated. Then, with respect to the
property vector, the mouth label is found which satisfies isosceles triangle
with the eyes. Required facial features are found which are right eye, left eye
and mouth. Face image can be extracted which covers two eyes and mouth.
Face covering is created with a rectangle in which covers eyes and mouth.
Facial feature extracted image from face candidate and face image.

This proposed algorithm is implemented using MATLAB and tested for more
than hundred images. This algorithm detects not only one face but also more
than one face. Small amount of oriented face are acceptable. Results are
satisfactory for all purpose.
 PROJECT STRUCTURE

The Project Aims to develop a system through which whenever a person comes
inside the frame of a CCTV Camera installed across the globe the system perform
a face detection algorithm and extract the face and saves it in a Database which
can be used to perform many different operations in future.

Some of the Operations that can be performed by the data are:

 Police Verification – Police Department can use the data for predicting the
criminal by matching the footage from the crime scene with the database.
 Location Tracking – Location of an individual can also be tracked using this
as the output data contains the time and location of the face that it extract.

This project is using many different Libraries for the operation

The Software used in this Project is:

 Spyder Anaconda IDE Software

The Programming Language Used is:

 Python 3.7

The Libraries used are:

 Os Library
 NumPy
 OpenCV
 Time
 DateTime
Hardware Used:

 WebCam- WebCam is used just for demonstration in actual it will be

replaced by a CCTV System.
 Computer System – A Computer is used to run the code and store the data
for further operations#
 Digital Display- Digital Display such as Computer Display or a TV screen can
be used to monitor the footage in real time.#

Note

#- This are optional which can be used but not used in this project.
 Project
Import libraries
I import os to access various files in the directory. Then, cv2 will be used to work
with images. numpy helps to easily work with multi-dimensional arrays.

Camera Access

This line of code is used to access the camera for detecting the face.

I am using webcam so the number is 0

If the camera access is made then the while return True Value and further code
will run.

Define paths and load model

The model’s prototxt and caffemodel is provided in the OpenCV repo itself. I used
the same and placed them in the model_data directory in my project. prototxt file
includes the text description of the network and caffemodel includes the weights.
I read the two files and loaded my model using cv2.
Create directory

If the directory where the resultant images will get stored does not exist, I’ll
create the directory. The output folder is faces.

If the directory where the resultant images will get stored does not exist, I’ll
create the directory. The output folder is updated_images.
Read images

I loop through all images inside the images folder. After extracting the extension, I
check that the files are either of the type .png or .jpg and just operate with those
files only.

Identify Each Face

One by one, I iterate over all of the faces detected in the image and extract their
start and end points. Then, I extract the confidence of detection. If the algorithm
is more than 50% confident that the detection is a face, I show a rectangle around
it. And the rectangle part which detect the face is cutout and saved in the
directory created as faces
Printing the output In the consol

The output file details which is saved in the directory is shown in the consol
The Entire Code for the Project is as under.
Software Download Links:
Python 3.7
https://www.python.org/downloads/

Anaconda Enviroment
https://www.anaconda.com/distribution/
 CONCLUSION
Face recognition systems are part of facial image processing applications
and their significance as a research area are increasing recently.
Implementations of system are crime prevention, video surveillance,
person verification, and similar security activities. The face recognition
system implementation will be part of humanoid robot project at Atılım
University.
Knowledge-Based face detection methods are used to find, locate and
extract faces in acquired images. Implemented methods are skin color and
facial features.
RGB color space is used to specify skin color values, and segmentation
decreases searching time of face images. Facial components on face
candidates are appeared with implementation of LoG filter. LoG filter
shows good performance on extracting facial components under different
illumination conditions. Proposed algorithm is capable of detecting
multiple faces, and performance of system is acceptable.
Proposed system can be affected by pose, presence or absence of
structural components, facial expression, imaging condition, and strong
illumination.
 REFERENCES
 https://en.wikipedia.org/wiki/Python_(programming_language)
 https://en.wikipedia.org/wiki/IDLE
 https://en.wikipedia.org/wiki/PyCharm
 https://en.wikipedia.org/wiki/Spyder_(software)
 https://en.wikipedia.org/wiki/Project_Jupyter
 https://en.wikipedia.org/wiki/Computer_vision
 https://en.wikipedia.org/wiki/OpenCV
 https://docs.opencv.org/master/
 https://github.com/kb22/Create-Face-Data-from-Images
 https://docs.python.org/3/library/idle.html
 https://towardsdatascience.com/extracting-faces-using-opencv-face-detection-neural-network-
475c5cd0c260
 https://www.google.com/webhp?hl=en&ictx=2&sa=X&ved=0ahUKEwjouqugm-
XoAhU9H7cAHd_1AcYQPQgH