International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-

nized by Karpagam Institute of Technology, Coimbatore

AI POWERED AUTHENTICATION FOR SMART

HOME SECURITY
Mrs. S. Suganya B.E, M.E,
Assistant Professor/CSE
Chettinad College of Engineering and Technology
Karur, India
suganya@chettinadtech.ac.in

Sriram .V Malishwaran .M
Computer Science Engineering Computer Science Engineering
Chettinad College of Engineering and Technology Chettinad College of Engineering and Technology
Karur, India Karur, India
sriiiram321@gmail.com mmalishwaran2002@gmail.com

Hariharan .T
Computer Science Engineering
Chettinad College of Engineering and Technology Eliza John .A
Karur, India Computer Science Engineering
hariharantamilselvan009@gmail.com Chettinad College of Engineering and Technology
Karur, India
johnkrp31@gmail.com

Abstract— The visitor management is a modern world
problem with its application a numerous fraud, privacy issues,
etc. can be quickly identified and avoided. The visitor
management system using face recognition is one of the most
secure systems even better than CCTV cameras and wake
through gate methods. The primary decision that needs to be
made in the project is whether the system's cost is consistent
with its scope. The size of operations and the security needs
vary depending on the location, for example, household use
versus industrial use. The majority of organizations, schools,
and colleges now employ visitor management systems, but
thanks to remarkable improvements, this can be expanded to
include railway stations, airports, toll stations, etc. Most
companies with large facilities integrate visitor management
systems into their total security, which is continually expanding
at a constant rate. It is suggested that the face recognition
visitors' management system (FRVMS) improve home security
by automatically identifying strangers. Having a centralized
system will make managing and monitoring processes more
effective. Cost of development is also taking into consideration
as this system is not requiring any extra devices. Face
recognition is performed utilizing the computer's built-in web
camera. The family of face data contained in the monitoring
system's database is compared with the detailed features that
have been detected and security is cancelled in case of a
member, while an alarm notification is displayed to the user in
case of an outsider. Then the user wear face mask means,
specify the alarm to remove the mask to recognize the facial
features. We can implement the framework using deep learning
algorithm named as Convolutional neural network and
experiment this system in real time environments.
Index Terms — Face recognition, security, deep learning, visitor
management, facial features.
I. Introduction
Visitor management has become a critical concern in the
modern world due to the increasing instances of fraud and
privacy issues. One of the most secure and efficient solutions
to this problem is the implementation of a face recognition
visitor management system (FRVMS). This system utilizes
deep learning algorithms, such as Convolutional Neural
Networks (CNNs), to automatically identify and verify
visitors based on their facial features. Unlike traditional
methods such as CCTV cameras and wake-through gate
methods, FRVMS eliminates the need for additional devices
and can be implemented using a computer's built-in web
camera. The system can be used in various settings, ranging

`
International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-
nized by Karpagam Institute of Technology, Coimbatore
from households to industrial facilities, schools, colleges,
railway stations, airports, toll stations, and other large
organizations. With a centralized database of face data,
FRVMS offers a more effective and efficient approach to
managing and monitoring visitors, improving overall security.
In this paper, we will discuss the benefits and applications of
FRVMS, including its potential to enhance home security and
its scalability to different environments. We will also
highlight the cost-effective nature of the system and its
potential for real-time implementation using deep learning
algorithms making it suitable for a diverse range of security
needs.
Furthermore, FRVMS offers enhanced security by
automatically identifying strangers and triggering alarms to
notify users in case of an outsider. Even if a visitor is wearing
a face mask, the system can prompt them to remove the mask
for facial recognition, ensuring accurate identification. The
use of deep learning algorithms, such as CNNs, allows for
robust and reliable face recognition, even in complex and
dynamic environments.
FIGURE 1. Security of smart home market prospect
The scalability of FRVMS is another notable advantage.
While t is already widely adopted in organizations, schools,
and colleges, it has the potential to be extended to various
other settings, such as railway stations, airports, toll stations,
and more. This scalability is attributed to the continuous
advancements and improvements in face recognition
technology, The cost-effectiveness of FRVMS is also a
significant factor to consider. Since it leverages existing
hardware, such as built-in web cameras in computers, it
eliminates the need for additional expensive equipment,
making it a cost-efficient solution compared to other security
methods. Moreover, by integrating FRVMS into an
organization's overall security system, it can streamline
visitor management processes and reduce the need for
manual monitoring, resulting in cost savings in the long run.
In conclusion, the implementation of a face recognition
visitor management system (FRVMS) using deep learning
algorithms offers a secure, scalable, and cost-effective
solution for managing and monitoring visitors in various
settings. With its ability to automatically identify visitors
`
based on facial features, even in challenging situations,
FRVMS can significantly enhance security and streamline
visitor management processes. As technology continues to
advance, FRVMS has the potential for widespread adoption
and can greatly contribute to addressing the modern world
problem of visitor management in a more efficient and
secure manner.
II. RELATED WORKS
The importance of face recognition systems for managing
visitors for tackling security issues has been highlighted by
prior research and related publications. Compared to other
techniques like surveillance footage and gate wake
mechanisms, face recognition offers a superior level of
security. The system is affordable, adaptable for broad
operational and safety needs, and has the ability to be
expanded to other sites. Through the use of built-in web
cameras, centralization improves management and monitoring
procedures while obviating the need for extra hardware.
Quick verification is made possible by face detection and
database comparison, with security being cancelled for
recognized members and alert notifications for outsiders. For
enhancing the system's accuracy, face masks and the
application of deep learning techniques like Convolutional
Neural Networks, have been suggested. Additional
investigation and experimenting may be helpful.
1) KNOWLEDGE-BASED AUTHENTICATION
An authentication technique utilised in powered by AI au-
thentication systems for secure smart homes is knowledge-
based authentication. It entails confirming a user's identifica-
tion based on their familiarity with specific data, like a pass-
word, PIN, or the solutions to security questions. Know-
ledge-based identification can be used as an additional layer
of protection in the context of security for smart homes to
guarantee that only authorised users can access and control
smart home systems and devices.
Users may be asked to enter a password or PIN when check-
ing in to a mobile app or web portal for using their smart
home controls in a knowledge-based authentication system
used for smart home security. They might also be asked to re-
spond to security queries they've already set up. during the
initial setup process. The answers to these security questions
International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-
nized by Karpagam Institute of Technology, Coimbatore
are checked against the pre-registered answers in the system
to verify the user's identity at the time of initial setup. To
confirm a user's identification, the responses to III. DESIGN FOR PROPOSED SYSTEM & DEEP
LEARNING
A. SYSTEM STRUCTURE
these safety concerns are compared to the pre-registered re- The overall structure of the system proposed in this paper
sponses in the system. is shown in a block diagram as shown in Fig.5. The
composition of the system is largely divided into parts in
charge of registration and processing. In detail, it consists of
2) OWNERSHIP-BASED AUTHENTICATION
(1) deep learning model training, (2) visitor face detection, (3)
Ownership-based authentication in AI-powered smart home
face recognition and access verification, and (4) monitoring.
security uses ownership-based credentials to verify users' au-
thenticity and authorization. It involves unique ownership- 1) DESCRIPTION OF EACH COMPONENT
based attributes associated with users or smart home devices, (1) Part of a deep learning model used for training (derived
verified using AI-powered techniques. from a deep learning training model): Desktop - Save pictures
of loved ones and acquaintances in a dataset beforehand and
3) OBJECT CHARECTERISTIC BASED use the Tiny-YOLOv3 deep learning model to learn them.
AUTHENTICATION (2) Component for collecting video data: Uses a webcam
Object characteristic-based authentication is a type of authentic- CCTV to gather visitor video.
ation that verifies users by their attributes. A representative ob- (3) A database that stores and processes the gathered video
ject characteristic-based authentication technique is biometrics. pictures serves as the control and processing component.
Biometrics separate physical and behavioural characteristics. (4) Monitoring and post-processing: The
Physical characteristics include fingerprints, iris, vein, and vein manager's smartphone receives the processing result and
recognition, as well as other techniques like retina and hand processes it there.
form, which are used to recognize faces using facial shapes and
thermal images of each individual. Speech recognition, gait re- 2) HARDWARE CONFIGURATION DIAGRAM
cognition, and signature recognition are examples of behavi- The operation of the entire system for face detection, using
hardware specifications
oural traits. Recently, businesses have started utilizing behavi-
Processor : Intel / Ryzen processor
oural characteristics like voice, gait, and signature recognition
RAM : 4 GB
to improve biometric accuracy [12], [13], [14]. Compared to
Keyboard : standard keyboard
voice, vein, iris, and fingerprint recognition, face recognition is
Monitor : 14 / 15.6 inch monitor
more practical. The accuracy of iris recognition is higher than
that of other biometrics, and installation costs are lower than
those of speech recognition systems.
Recently, a permanent and inexpensive fingerprint re-
cognition system or a Face ID system based on the user's face
have been implemented and used nearly universally to identify
the phone's owner and enable smartphone use and access.

4) MULTI-FACTOR AUTHENTICATION
knowledge-based, or password-based, authentication is vulner-
able Because it permits using short, insecure passwords that are
frequently never changed from the password that was used to
register, knowledge-based or password-based authentication is
weak. To circumvent the weakness of passwords, extra authen-
tication measures are therefore required. By first verifying us-
ing their ID and password and then utilizing a different addi-
tional authentication method, users of multi-factor authentica-
tion can increase security. Ownership-based authentication and
object characteristic-based authentication are further multifactor
authentication strategies. With capabilities like fingerprint, face,
OTP, and random number input through SMS, cell phones in
particular are being utilized as a way of authentication.

`
International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-
nized by Karpagam Institute of Technology, Coimbatore
IV. STRUCTURE AND PROTOCOL OF THE
PROPOSED SYSTEM
A. PROTOCOL:
An artificial intelligence algorithm known as Algorithm 1
uses the CCTV camera that is mounted on the front door to
identify visitors. It will notify the user to inspect the visitor's
face if it cannot recognize them after a great distance or while
they are wearing a mask. It takes a picture of the visitor
during this process to record their visit. When it recognizes a
visitor's face and determines that they are a family member, it
unlocks the door. It asks for additional identification if the
initial identification is unsuccessful.
[Step of visitor recognition]:
1. The visitor could be a member of the family or a guest
who is just dropping by a bystander at the front door, or a
courier bringing something. The guest is only recognized after
spending a specified amount of time within the proximity
sensor's range while standing at the front door. The system
then exits standby mode and begins to work.
2. The system uses the web camera to snap a photo of the
visitor in order to identify them, and the learning model
developed using deep learning is used to identify the visitor's
face.
3. Following the face detection, the system snaps a photo
of the face's centre and saves it. This is kept so that data about
suspect individuals can be gathered.
Class name Class number
Eye_area 01
Mask 02
Eyebrow 03
Face(head) 04
R_eye 05
L_eye 06
mouth 07
[Step of visitor decision]:
1. It could be a guest entering the home or a delivery
person leaving a package at the front entrance. A family
member or guest can ring the doorbell or enter the password
for the door lock if they want to enter the house. Based on
how long a person waits at the front entrance, the algorithm
determines if they are a customer or a visitor. The doorbell is
pressed to let in guests, and the door lock password is pressed
to let in family members.
2. The system determines whether the visitor is a guest or
a member of their family. The model locates a portion of the
visitor's face to identify in order to check the visitor's face.
The system will produce the message "Please come closer" or
give that message through the speaker if it is unable to detect
that part of the face.
`
3. If the visitor's facial region is discovered, the visitor's
face image is saved.
[Step of face recognition]:
1. Recognize a face in the CCTV image.
2. Check whether a mask is being worn if a face is not
being recognized, and if it is because of a mask, output the
message "Please pull down your mask" or say it aloud.
3. If a mask is not being worn, go back to finding the face
area and taking another photo of it.
4. If the face is now recognized, see if a family member
who is stored in the database is present.
[Steps for post-authentication process]:
1. Open the front door automatically and use easy
authentication to verify the visitor is a registered family
member.
2. The manager receives the visitor's face image in the
case of a guest. After verification, the manager may then
unlock the front door. If the system is unable to identify a face
registered in advance, the visitor may be authenticated before
the front door is opened by entering the door lock password.
In the suggested system, the front door is opened by a device
that inputs the password through a touchpad when the correct
password, which was previously stored, is input. When the
password typed on the touchpad as a string of numbers
matches the password that has been previously recorded, the
front door lock system authenticates guests.
[Steps for CCTV image capturing]:
1. Using CCTV, faces are found in the image using a
model that has been trained in advance with deep learning.
2. The number of faces in the recorded image is
determined.
The system determines whether one visitor is peeking if
there are two or more visitors.
Attack by shoulder surfing? It is assumed that the person
in the back is snooping if they approach the front door or peer
over the front visitor's shoulder at the door lock. The alert
message "Please enter the password carefully so that no one
else can see" will then appear on the door lock display. After
that, the user defends against this social engineering assault
by carefully typing the password.
3. The message "Please input the password" is displayed if
there are more than two visitors but no peeking is found, or if
there is only one visitor.
[Steps for password input]:
1. The visitor inputs the password on the touchpad.
2. If the password entered matches the password stored in
the database, the front door opens.
There are numerous assaults that might be made against
this system because it uses face recognition to decide whether
International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-
nized by Karpagam Institute of Technology, Coimbatore
or not to unlock the door lock. Users can, for instance, be
verified using pictures of their relatives. As a result, the
suggested system recognizes when feature points on a face
change and determines whether or not the face is a photo, in
which case user authentication is prohibited.
3. The process of employing CCTV at the front door to
verify that a guest is a family member. The suggested model
uses a deep learning-trained model to identify a visitor's face
in a CCTV image. When a face image captured by CCTV is
compared to the face data of family members who have
previously registered, and the comparison value exceeds the
predetermined threshold, authentication is complete.
Following that, the subsequent step is carried out after the
visitor information is recorded in the database access log
table. If more authentication is needed, the front door lock can
be unlocked by entering the door lock password, which
completes authentication.
V. ANALYSIS AND EXPERIMENTAL RESULTS
OF THE PROPOSED SYSTEM
A. ANALYSIS OF USER AUTHENTICATION
SYSTEM
When the visitor’s face is recognized with a CCTV image,
81 feature points are detected using the model that was
trained on the file shape_predictor_81_face_landmarks.dat. A
linear classifier is used to locate facial landmarks including
the eyes, nose, mouth, chin line, and eyebrows using the HoG
feature as a face detector. The faces of registered users or
family members are already saved in the database for user
authentication. The familyTbl table in the database contains
the vector values of the feature points in the family members'
face photos. This function collects the feature point vector
values from the visitor's face image and compares them to the
feature point vector values of family members that are kept in
familyTbl. When the threshold value falls inside the
acceptable range for authentication, authentication is carried
out.
part of Python code that extracts 81 facial features. To
extract and store features from the family photo, it makes use
of the Dlib library. The feature in the video image captured by
the webcam when a visitor is waiting at the main door is
extracted by the source code, and it then verifies that the
image represents the same person by comparing the vector
value of the previously saved feature of the family to the
extracted vector value.
B. EVALUATION OF ANALYSIS OF SECURITY
FIELD
The following conditions are met by the requirements
specified in the introduction.
First, based on faces discovered by CCTV, the suggested
model can be used to count the number of visitors. After a
`
specified amount of time, it is determined that the group of
people is a passerby if its size reduces. If not, it is assumed to
be a guest or an intruder.
Second, the person behind the visitor can sneak a peek if
the visitor at the front touches the password for user
authentication. The guest who is standing behind the lead
person must therefore be identified as either a family member
or a companion.
The CCTV outputs a message to the front visitor on the
display alerting them of this if the visitor's face is perceived as
approaching the door or if the CCTV detects that the visitor's
eyes are attempting to steal the password. As a result, when
entering a registered password, the visitor in front is informed
that their password is too short if the person behind is
determined to be peeping.
Third, relatives register ahead of time. It gains knowledge
from the stored images, assesses whether a face has been pre-
registered from the video image entered into the CCTV, and
verifies the visitor's identity.
By matching the CCTV image with a previously recorded
face, an authentication result is generated probabilistically. If
the calculated result exceeds the threshold, primary
authentication is carried out; however, if it is less than 1,
secondary authentication is necessary due to the possibility of
incorrect authentication. Although secondary authentication
can be accomplished in a number of ways, in this study it can
be done by entering a brief password. The door is opened
once this authentication is finished.
Fourth, the manager's phone can receive the CCTV image
in the proposed model via the smart home service. The
manager can see if someone is there by looking at the image.
The guest cannot enter the residence with authentication if the
password authentication fails or if a family member's visit is
not recorded.
In this instance, it uses a smartphone to check the guest
before remotely opening the door. The important image is
saved on the server and the verified visitor's name is recorded
in a log file as a result of the suggested procedure. When
members of the family return home, access information, use
pattern analysis, or have a prolonged absence, this log file
may also offer extra services.
VII. CONCLUSION
Deep learning, a key artificial intelligence technology, is
expanding quickly as it is used for speech and picture
identification. Particularly, autonomous driving and crime
prevention monitoring systems, which are emerging as future
industries, are using deep learning technology in the field of
image recognition as a core technology. Regarding the use of
deep learning models for image recognition, there have been
International Conference on Latest Trends in Science, Engineering and Technology (ICLTSET’23) on May 5&6, 2023 orga-
nized by Karpagam Institute of Technology, Coimbatore
many proposed algorithms that have enhanced and developed
CNNs that can process images.
We introduce several object detection algorithms,
including CNN, in this study. A visitor's face is detected by
CCTV, which then recognizes 81 feature points to produce a
collection of vector values depending on the features. Family
members have face images registered in advance. In this
work, we built and put into practice a system that, upon
identifying a new visitor as a member of the family, unlocks
the front door when the gap between the recognized facial
feature vector values between the CCTV image and the image
stored in the database is smaller than the predetermined
threshold.
Future study will focus on creating and researching a
system that can identify faces while regulating entry to stores
or restaurants, automatically track repeat customers, count
visitors, identify the amount of visitors, store visitor
information, and regulate access based on visitors.
Additionally, a system for access authentication must be
created that has a low false detection rate and can enable
simple access using face recognition rather than fingerprints
or passwords.
REFERENCES
[1] Tsai, Tsung-Han, et al. "Design of wireless vision
sensor network for smart home." IEEE Access 8 (2020):
60455-60467.
[2] L. L. Chambino, J. S. Silva, and A. Bernardino,
‘‘Multispectral facial recognition: A review,’’ IEEE Access,
vol. 8, pp. 207871–207883, 2020
[3] Waseem, Muhammad, et al. "Face recognition for
smart door lock system using hierarchical network." 2020
International Conference on Computational Intelligence
(ICCI). IEEE, 2020.
[4] Li, Huoyou, et al. "UFaceNet: Research on Multi-Task
Face Recognition Algorithm Based on
CNN." Algorithms 14.9 (2021): 268.
[5] Lee, Min-hye, and Hyung-Jin Mun. "Comparison
analysis and case study for deep learning-based object
detection algorithm." Int. J. Adv. Sci. Converg 2.4 (2020): 7-
16.
[6] Jiang, Xinbei, et al. "Real-time face mask detection
method based on YOLOv3." Electronics 10.7 (2021): 837.
[7] Shah, Syed W., and Salil S. Kanhere. "Recent trends in
user authentication–a survey." IEEE access 7 (2019): 112505-
112519.
[8] Renuka, K. M., et al. "Design of a secure password-
based authentication scheme for M2M networks in IoT
enabled cyber-physical systems." IEEE Access 7 (2019):
51014-51027.
[9] Fang, Wei, Lin Wang, and Peiming Ren. "Tinier-
YOLO: A real-time object detection method for constrained
environments." IEEE Access 8 (2019): 1935-1944.
[10] M. Alsawwaf, Z. Chaczko, M. Kulbacki, and N.
Sarathy, ‘‘In your face: Person identification through ratios
and distances between facial features,’’ Vietnam J. Comput.
Sci., vol. 9, no. 2, pp. 187–202, May 2022.