Face and Features Landmark Detection in Low Light Condition

Anand Upadhyay*
Department of Information Technology
Thakur College of science and commerce,
Mumbai, 400101, India.
Email: anandhari6@gmail.com
*Corresponding author

Nidhi Parikh
Department of Information Technology
Thakur College of science and commerce,
Mumbai, 400101, India.
Email: immsnidhi@gmail.com

Chinmayee Mhaskar
Department of Information Technology
Thakur College of science and commerce,
Mumbai, 400101, India.
Email: chinumhaskar796@gmail.com

Abstract: - This Paper represents the face detection using advanced method deep
neural network which uses deep learning frame work. The old models used to
detect the faces were like Haar-cascade method which detect the faces with good
approaches but there is some uncertainty in the accuracy of the old models, so in
this system we will use the latest deep neural network model which is embedded
with latest open cv and by using the deep learning model frame work which is
weighted with some other files. In this it is not only facial Detection, but also
recognizing the parts of the face, eye, mouth in low light after designing programs
in addition to the new algorithms with the theory of CaffeModel and Dldetector.
The benefit of this work emerges in the areas of security, airports, markets, and so
on.

Keywords: - Deep neural network, dnn, caffe model, face detection, face features,
recognition, low light, landmark detection, Dldetector.

Reference to this paper should be made as follows: Anand Upadhyay, Nidhi Parikh,
Chinmayee Mhaskar, 2022, “Face and Features Landmark Detection in Low Light
Condition” Int. J. Image Mining, Vol. 4, No. 1, pp.114-123.

Biographical notes: Anand Upadhyay received his Master of Science in Information

Technology from University of Mumbai, Mumbai, MCA from Annamanlai University
M.Tech (ALCCS) from IETE, New Delhi India. He is working as an Assistant
Professor in the Department of Information Technology, Thakur College of Science
and Commerce, Maharashtra, India since 2012. His research interest includes pattern
recognition, machine learning and image processing, neural network.

Nidhi Parikh completed her UG degree in 2021. She is presently pursuing PG

degree in Information Technology at Thakur College of Science and Commerce,
Mumbai, India.

1
 Chinmayee Mhaskar completed her UG degree in 2021. She is presently pursuing
PG degree in Information Technology at Thakur College of Science and
Commerce, Mumbai, India.

Introduction: -

Face detection plays a very important role now a days in various

applications. First, we need to know about the difference between the face
detection and recognition. Face detection is nothing but detecting the face
of any person present, if there is a face, first try to detect the face features
and then entire face, and face recognition is an application of face
detection. Facial detection is one of the most popular topics in the deep
learning field for object detection.
There are hundreds of facial detection algorithms today, but the first facial
detection algorithms were developed in the early seventies. Since then, the
accuracy of the algorithms has improved so much, that nowadays face
recognition is often used over the biometrics, often as a part of (or together
with) a facial recognition system. It is also used in video surveillance,
human computer interface and image database management.
The main aim of this paper is to achieve the better accuracy in low light
face detection rate by using the deep neural network which uses neural
network as its base for processing inputs which contain hidden layers and
by using the deep learning frame work which comes with Caffe Model
which is pre-trained. This will use the camera of the computer or the
system that would detect and recognize the person’s face and face features
landmarks of the individual using the tool in OpenCV called as the Open
Face, Dldetector and python programming language in deep neural
network.

Literature Survey: -

Ge Wen, Huaguan Chen, Deng Cai, Xiaofei He. Description: A model

named as CNN has been proposed in this paper for face recognition using
OpenCV Convolutional Neural Network (CNN), LFW, FaceNet
architecture which is used as an image classifier. OpenCV used in CNN
contains SSD with ResNet-10 as backbone and is capable of detecting
faces in most orientations. We achieved 99.33% on the LFW benchmark
with only single CNN model and similar performance even without face
alignment [1]. Kumar Y, Sharma S Description: In their paper, firstly it is
explained about the process of Face Expression Recognition (FER) which
further assisted that how a particular technique is modelled. Also, an
extensive survey of the latest techniques for FER has been presented
comparing them based on their feature extraction technique, classification
method and recognition rate [2]. Dhavalikar AS, Kulkarni RK.
Description: In this paper, an Automatic Facial Expression Recognition
System (AFERS) has been proposed. The proposed method has three
stages: (a) face detection, (b) feature extraction and (c) facial expression
recognition [3]. Anirudha, B Shetty, Bhoomika, Deeksha, Jeevan, Rebeiro,
Ramyashree. Description: This project proposed idea about how the home
security can be enhanced by using an algorithm for face detection and
recognition (Haar Cascade Classifier). The Haar Cascade Classifier model
is used with KNN Algorithm, LBP. [4] P.Anjaneyulu, S.V.S Prasad,
V.Syambabu, A.Vamshi Kumar. Description: This Paper represents the
face detection using advanced method deep neural network which uses
deep learning frame work. This model detects the faces accurately and

2
 paves the way for better recognition systems which can be used in several
face biometric applications [5].

Methodologies: -

3.1. Deep Neural Network

A neural network, in general, is a technology built to simulate the activity

of the human brain – specifically, pattern recognition and the passage of
input through various layers of simulated neural connections.
Subsequently, G. Thomson had proposed a new idea known as deep
learning, where it is a new model training as shown in Figure 1.

Figure 1. Deep Neural Network

Deep neural networks are networks that have an input layer, an output
layer, and at least one hidden layer in between. Each layer performs
specific types of sorting and ordering in a process that some refer to as a
“feature hierarchy.” One of the key uses of these sophisticated neural
networks is dealing with unlabelled or unstructured data. The phrase “deep
learning” is also used to describe these deep neural networks, as deep
learning represents a specific form of machine learning where technologies
using aspects of artificial intelligence seek to classify and order
information in ways that go beyond simple input/output protocols.

3.2. Face Detection

Face detection is a computer technology being used in a variety of

applications that identifies human faces in digital images. Face detection
also refers to the psychological process by which humans locate and attend
to faces in a visual scene. Face detection can be regarded as a specific case
of object-class detection. In object-class detection, the task is to find the
locations and sizes of all objects in an image/frame that belong to a given
class.

The goal of face detection is to determine if there are any faces in the
image or video. If multiple faces are present, each face is enclosed by a
bounding box and thus we know the location of the faces. Human faces are
difficult to model as there are many variables that can change for example

3
 facial expression, orientation, lighting conditions and partial occlusions
such as sunglasses, scarf, mask etc. The result of the detection gives the
face location parameters and it could be required in various forms, for
instance, a rectangle covering the central part of the face, eye centers, or
landmarks including eyes, nose and mouth corners, eyebrows, nostrils, etc.

The OpenCV DNN module in DNN only supports deep learning inference
on images and videos. It does not support fine-tuning and training. Still,
the OpenCV DNN module can act as a perfect starting point for any
beginner to get into the field of deep-learning based computer vision and
play around. One of the OpenCV DNN module’s best things is that it is
highly optimized for Intel processors. We can get good FPS when running
inference on real-time videos for object detection and image segmentation
applications. We often get higher FPS with the DNN module when using a
model pre-trained using a specific framework.

Steps for Face detection:

When using OpenCV’s deep neural network module with Caffe models,
you’ll need two sets of files:

• The prototxt file which define the model architecture (i.e., the
layers).

• The caffemodel file which contains the weights for the actual
layers. Both files are required when using models trained using
Caffe for deep learning.

• We initialize a VideoCapture object specifying camera with index

zero as the source (i.e., laptop’s built-in camera or your desktop’s
first camera detected).

• We allow the camera sensor to warm up for 2 seconds and we

loop over the frames and compute face detections.

• Compare to the confidence threshold and draw face boxes +

confidence values on the screen.

Figure 2: Flowchart for Face Detection

4
 Landmark detection:
The facial landmark detection tells all the required features of a human
face which we want. The face detector is the method which locates the
face of a human in an image and returns as a bounding box or rectangle
box values. After getting the face position in an image and next we find
the small features of the faces like eyes, eyebrows, lips, etc. The
landmarks can be roughly split into two different categories: facial
keypoints and interpolated landmarks. The facial key points are the main
landmarks like the corners of the eyes or mouth whereas the interpolated
landmarks connect the key points and/or describe the shape of the region
of the face. Figure 1 shows an example annotation of a face with 68
landmarks.

Figure 3: Annotation of a 68-point face features landmarks.

Steps for Facial landmark detection:

• We will be using same model (Caffe model) for this.
• Take the frames from the VideoCapture then preprocess those frames by
resizing and
• converting them to grey scale.
• Localize the frame through detector which will return the bounding box (x,
y coordinates) for each frame.
• We have detected the faces in the video stream, the next step is to apply
the facial landmark predictor to each face.
• Display the output frame that shows the landmark for face and the face
features.

5
 Figure 4: Flowchart for Facial landmark detection

3.2. Accuracy: -
Confusion Matrix calculates the number of correct & incorrect predictions
which is further summarized with the number of count values and
breakdown into each classis. It can be used to get precision, accuracy,
recall etc. Confusion matrix is provided by the metrics module of the
sklearn. Table II shows the confusion matrix with TP, FP, FN, TN.

Table I: Confusion Matrix

The accuracy of the system will be tested via recognition of three peoples
with multiple times at different locations, mainly to test how light intensity
affect the accuracy of the system. The accuracy is verified using confusion
matrix. The calculation is based on (1).

((TN + TP) / Total) x 100% ……..(1)

where TN is true negative while TP is true positive.

Experimental results: -

4.1. Face and landmark detection in normal light condition:

6
 A1 A2

Figure 5: A1& A2 are face and landmarks detected frames with

accuracy 99.75% and 99.64%.

4.2. Face and landmark detection in low light condition

B1 B2

Figure 6: B1 & B2 are face and landmarks detected frames with

accuracy 88.68% and 94.42%.

4.3. Face and landmark detection in dark light condition

7
 C1 C2

Figure 6: C1 & C2 are face and landmarks detected frames with

accuracy 98.94% and 98.29%.

Here, we get average accuracies of 99%, 91%, and 98% for the above-
mentioned light conditions. From this, we came up with, the dnn algorithm
is best for face detection and features, landmark detection in dark light
conditions and predicting faces in parking areas and for other security
purposes.

Conclusion: -

Face recognition sy1stems are currently associated with many top

technological companies and industries making the work of face
recognition easier. The use of python programming and OpenCV makes it
an easier and handy tool or system which can be made by anyone
according to their requirement. The proposed system discussed in this
paper will be helpful for many as it is user friendly and cost-efficient
system. In this we have tried to achieve better accuracy of different age
group, different light conditions, and different distances from the screen.
Hence, by the use of python and OpenCV the face recognition system can
be designed for various purposes.

References: -
Z. Suna, G. Bebisa, R. Millerb, March 2004, Object detection using
feature subset selection, Elsevier, the Journal of Pattern Recognition, p.
581-589.

Mabrook S. Al-Rakham, Jakaria Rabb, Tahmid Hasan Fuad and Awal

Ahme, 2021, Recent Advances in Deep Learning Techniques, in IEEE
Access, vol. 9, pp. 99112-99142.

S.KevinZhou, 2016, Introduction to Medical Image Recognition,

Segmentation, and Parsing, pp. 1-21

Anirudha, B Shetty, Bhoomika, Deeksha, Jeevan, Rebeiro, Ramyashree,

2021, Facial recognition using Haar cascade and LBP classifiers, vol. 2,
pp. 330-335.

8
 Shilpi Singh, S.V.A.V. Prasad, 2018, Techniques and Challenges of Face
Recognition: A Critical Review, vol. 143, pp. 536-543.

Dhavalikar AS, Kulkarni RK, 2014, Face detection and facial expression
recognition system, pp. 1–7.

Kumar Y, Sharma S 2017, A systematic survey of facial expression

recognition, pp. 1074–1079.

Ge Wen, Huaguan Chen, Deng Cai, Xiaofei He, 2018, Improving face
recognition with domain adaptation, vol.287, pp. 45-51.
Gao Y., Qi Y., 2005, Robust visual similarity retrieval in single model face
databases. Pattern Recognition, vol. 38, pp. 1009-1020.
Andreas Maier, Christopher Syben, Tobias Lasser, Christian Riess, 2019,
A Gentle Introduction to Deep Learning in Medical Image Processing, vol.
29, pp. 86-101.