SMART INTERACTIVE MIRROR WITH VOICE ENABLED CONTROL FOR HOME

AUTOMATION USING GOOGLE ASSISTANT

Hariharan P.S [1], Arunkumar A.M [2], Jagadeesh Chandra C.M [3], Darshan M.M [4]
,Saravana Kumar [5]
[1][2][3][4]
Final Year Students Department Of CSE , DSATM
[5]
Associate Professor Department Of CSE , DSATM
surveillance cameras to robotized pool controllers [2] [3]. Because
of its minimal effort, such undertakings are exceptionally well
known in instruction, where understudies can get occupied with
Abstract—Internet of Things, Smart Home, and hands-on ventures.
Smart Cities have become intriguing issues as of
late. The improvement of these zones becomes In this paper, the creators built up a shrewd mirror sys-tem. As
spirally because of the rise of cutting edge shrewd another expansion to the shrewd home family, brilliant mirrors
gadgets. Savvy mirrors are another expansion to increased expanding consideration as of late. There have been some
fascinating undertakings distributed on the Internet [4]. For
the brilliant home family that has been getting a
example, Philips HomeLab [5] made an astute individual
ton of consideration these days by both business consideration framework that executed an Interactive Mirror [6] to
producers and the scholarly community. This give altered administrations to clients. The framework is equipped
paper portrays how a Raspberry Pi gadget can be for showing TV encourages, screen the most recent climate, etc. The
utilized to improve such mirrors with insight and mirror has a LCD show joined with a reflected surface and a
security. The objective is to build up a financially processor to give the expected administrations. Another
undertaking created by Sam Ewen and Alpay Kasal at Lit Studios [7]
savvy smart mirror framework that functions as
is a touch and signal useful mirror. The fundamental technique for
an ordinary mirror, yet in addition have the connecting with the mirror is by contacting the screen. Five
option to show different sorts of data, for understudies at Chalmers University in Sweden
example, climate, time and area, recent
developments, and clients. The mirror can give
sight and sound administrations while
guaranteeing top of the line security over the
whole framework .And home apparatuses, for
example, lights, fans can be controlled through
the voice directions utilizing Google colleague.

File Terms—shrewd mirror, Raspberry Pi, counterfeit intelli-

gence, voice acknowledgment, face acknowledgment, Internet of
Things, Alexa voice administration

I. INTRODUCTION

Brilliant Home has become a rising subject as of late in both

scholarly community and industry. With the exception of the well-
created business Internet of Things (IoT) items, for example, keen
entryway locks, savvy fittings, and brilliant indoor regulators, shrewd
home has pulled in countless individuals to chip away at self-
persuaded ventures utilizing minimal effort equipment, for example,
Raspberry Pi [1]. The known Raspberry Pi based brilliant home
ventures spread an assortment of themes, from Raspberry Pi fueled
 Segment IV exhibits the exploratory examination, and an end
is given in Section V.

designed the HUD Mirror [8]. They used a two-way mirror to
allow the LEDs mounted behind to illuminate the information.
The system can display information such as time, weather,
temperature, and a toothbrush timer.
The savvy reflect portrayed in this paper contrasts from the
previously mentioned undertakings mostly in that it offers
exceptional highlights to improve the framework security through
biometric authentica-tion, a voice right hand with included Alexa
abilities and sight and sound capacities. The biometric confirmation
is actualized with multi-variables, facial and voice, to guarantee a
higher security level than that gave by just voice which has been
created in Amazon Alex and Google Home. Additionally, re-
verification is authorized if the client has not been collaborating
with the mirror for some time. In this way, it can shield the
framework against a few conventional security dangers, for
example, session commandeering or man-in-center assault. Also,
the voice partner and the mixed media show actualized in the
mirror cause clients to appreciate the insight of the savvy home.
The guide of this paper is given as pursues: Section II
depicts the review of the framework. Segment III talks about
the subtleties of the framework structure and execution.
II. SYSTEM OVERVIEW
The savvy reflect framework is intended to be an easy to
understand arrangement that offers disentangled and customized
benefits as a mirror. Consequently, the framework was organized to
give the client an unlimited oversight over all administrations and
capacities through voice communication. Figure 1 delineates a
schematic diagram of the framework, which can be separated into
three unmistakable parts:
Physical structure
Smart mirror software
Cloud services
The physical structure of the framework comprises of a single
direction acrylic reflect put over a showcase screen, a Raspberry
Pi gadget, and a mouthpiece. The arrangement permits the
impression of light beams from one surface of the mirror while
enabling light from the presentation to go through. So a client
can see his refection notwithstanding the substance appeared
on the showcase. The arrangement is at last encased with a
wooden edge to give the regular mirror feel to the framework.
Weather and

Location Weather and location

API

News
Headlines

RSS News Feeds

A
u
t
Req h
e
uest n
ti
c
a
ti
o
n
F
r
a
m
e
w
o
r VoiceAssista
k nt
Amazon
Native Skill Webservices

AWS Lambda
Custom
Custom Skill Webservices

Host

Response

Processed Response to Custom Skill Request


The keen mirror programming was created utilizing Python
programming language. It comprises of a few modules which
handle singular elements of the framework, including climate
data, current city, time and date data, news sources and sight
and sound presentation. The mirror shows white characters over
a dark foundation for expanded perceivability from the opposite
side of the mirror.
The physical structure of the framework comprises of a single
direction acrylic reflect put over a presentation screen, a
Raspberry Pi gadget, and a mouthpiece. The arrangement
permits the impression of light beams from one surface of the
mirror while enabling light from the showcase to go through. So
a client can see his refection notwithstanding the substance
appeared on the presentation. The arrangement is at long last
encased with a wooden casing to give the regular mirror feel to
the framework.
III. SYSTEM DESIGN AND IMPLEMENTATION
This area subtleties how the framework is structured and
devel-oped. Initial, two key segments of the framework,
verification system and voice-based direction, are exhibited.
At that point the whole keen mirror framework is portrayed.
Figure 3 shows the structure of the system.in an image. Face
preparing is the way toward examining a few photos of a
similar individual to become familiar with the highlights
explicit to the person. This procedure makes a database of
the known clients. Face forecast is the way toward
distinguishing a known or pre-enlisted individual in another
picture. With the help of OpenCV, two modules, to be specific
client enrollment and client confirmation, were created in the
verification system. Figure 4 shows the general methodology
of face acknowledgment and enlistment.
All the above three figures represent the already
existing model.
1st figure represents the embedding working of a smart
mirror assemble
User Registration - face detection and training
This module was actualized by taking pictures with a
Raspberry camera. The pictures are passed into an OpenCV
course classifier to distinguish human countenances. A falling
clas-sifier is a model that is pre-prepared with a few hundred
face pictures, called positive models, and negative models -
discretionary pictures of a similar size. An example code bit is
appeared as pursues:
//Import opencv Library
import cv2
//Loading required cascading classifiers
face = cv2.CascadeClassifier(‘frontalface.xml’) //Load image and
convert to grayscale mode
img = cv2.imread(‘image.jpg’)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) //Face
detection in converted image
faces = face.detectMultiScale(gray, 1.3, 5) //Save image
containing faces to disc cv2.imwrite(‘image’+ str(Id))
On the off chance that a face is recognized in a picture
outline, the casing is spared. This progression is rehashed
until 20 photos of the client are spared and entitled with a
one of a kind whole number and the picture check (going 1 -
20). A one of a kind whole number is produced for every
client in the framework. At that point the preparation
procedure begins. All the spared pictures are stacked into an
exhibit, named faces, and went into a recognizer. The
recognizer is made as an occasion of the OpenCV Local Binary
Patterns Histograms class. Neighborhood Binary Patterns
(LBP) depicts a strategy for extricating valuable highlights
from face pictures. Fundamentally, this technique condenses
the nearby structure in a picture by contrasting every pixel
and its neighbors.
//Create a LBH face recognizer object recognizer =
cv2.createLBPHFaceRecognizer() //Train images and extract
face features //Each user has a unique integer Id
recognizer.train(faces, np.array(Id))) //Save extracted features
to face database recognizer.save
A. Voice Based Command
The framework underpins communication among client and
framework. Clients can control all capacities utilizing voice
directions, includ-ing enrollment, confirmation, looking, and
marking out. The usage highlights three essential advancements:
Google's "Discourse to-Text [12], Google's "Custom Search Engine
[13], and Amazon's "Alexa Voice Service (AVS)" [14].
Wake Word Detection
Like other voice-controlled frameworks, this venture actualizes
the utilization of a wake word. A wake word is a word or expression
that is articulated by the client to inform the framework that he is
prepared to give a guidance. The, endless supply of the wake word,
starts to sit tight for guidelines. The keen mirror utilizes the
expression "shrewd mirror" as the wake work. When the shrewd
mirror recognizes the wake word, it tunes in for the accompanying
guidance, for example, login, logout and so on.
Pass phrase Detection
Pass phrase detection refers to accepting and validating a
voiceprint from the user. This happens before user
enrollment and user authentication.
Registration Module
The enlistment module works somewhat unique in relation to the
two cases depicted previously. This module gives the client a brief
and sits tight for a reaction. The module initially requests that the
client choose and state his username. It at that point tunes in to the
username before mentioning a book translation from the Speech-to-
Text API. The client is provoked to affirm if the content translation is
precise after which the framework sits tight for a reaction, either
"yes" or "no". Once more, a book translation of the clients reaction
is gotten. On the off chance that it is "yes", the framework continues
to finish the enrollment. On the off chance that it is "no", the
framework comes back to the initial step to gather username.
Google Custom Search Engine: it is a free stage that empowers
designers to perform modified pursuits dependent on Google
Search. This administration likewise enables clients to penetrate
down to just those genuine site pages that are pertinent. With a
functioning Google account, clients can make a free custom web
crawler by visiting the Google custom hunt
Https://cse.google.com/cse to get an API key and custom internet
searcher ID. A Python piece is told underneath to show the best way
to question a custom web crawler.
query = ‘python programming language’; sv =
build(‘customsearch’, ‘v1’,
developerKey = key);
result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType =
‘png’,’).execute();
The above code scrap makes an exhibit called "result"
which contains the URLs to five pictures of PNG type that
match the depiction of Python programming language. This
undertaking utilized a custom web search tool to get sight
and sound substance dependent on client voice search
inquiries, which is talked about in detail in the following
segment
Amazon Alexa Voice Service (AVS): AVS is a plat-structure that
Amazon has given to permit engineers incorporate "Alexa" into
their tasks. With an AVS empowered item, clients can stream
media, get refreshes on climate, traffic, and news, just as ask
general information inquiries. Alexa (the cerebrum behind the
AVS framework) is activated when a client says the wake word,
"Alexa". When the gadget identifies the wake word, it starts to
tune in for a solicitation. The solicitation is then handed-off to
cloud based voice administration. It is along these lines a
necessity to have a working Internet association. The cloud-based
voice administration deciphers the solicitation and sends an order
back to the gadget or for this situation, the savvy reflect
framework.
 Extra/custom abilities - Skills are directions that Alexa can
process. They are basically applications however custom-made
explicitly for AVS. Alexa gives a wide scope of custom aptitudes
that are promptly accessible to clients of Alexa empowered
gadgets. Maybe the most amazing ability of AVS is that it
permits the formation of custom aptitudes that can be utilized
to improve the extravagance of client experience. The quantity
of existing custom abilities runs into the many thousands

This task made another Alexa ability called "Shrewd mirror". This
name speaks to what the utilizations state to show that they need
to collaborate with the expertise. The expertise is made by visiting
https://developer.amazon.com/alexa-abilities unit and choosing to
make another Alexa aptitude and posting the aims for the new
expertise. Plans are the different reasons/aims a client could have
Smart Mirror Software Display
for conjuring the expertise. When the goals have been determined,
the clients need to make a cooperation model. A collaboration
model speaks to how clients communicate with the new ability. The
B. Smart Mirror Software
model contains different proclamations a client may state to
cooperate with every one of the goals in the new expertise. The shrewd mirror programming is created utilizing python
master gramming language and PyQt 4 [17]. PyQt is a Python
When the model has been fabricated, the Alexa gadget (e.g., the authoritative of the cross-stage GUI toolbox Qt, executed as a
brilliant mirror) associated with the amazon engineer record would Python module. The savvy reflect has a presentation that is
have the option to get to the aptitude. An ability named "keen comprised of four unmistakable segments as enclosed Figure 10
mirror" for example may contain a plan called "getTime" and an
example articulation like "disclose to me the time". In the event that 1) Section A: This section displays weather and location
a client says "Alexa, request that shrewd mirror disclose to me the information. The data is obtained from two APIs, namely
time", the Alexa Voice Service would perceive that the client is freegeoip [18] and openweathermap [19]. Below is a code
conjuring the brilliant mirror ability. snippet showing how to extract data using the APIs.

//Call API to fetch city and region name

url = ‘http://freegeoip.net/json/’ + ip; r =

requests.get(url); js = r.json();

//Parse response to obtain city and region response =

(str(js[‘city’]) + ‘,’ +
js[‘region_name’]);

//Call API to fetch temperature and weather conditions

//Data retrieval using developer ID (APPID)
r= requests.get(‘http://api.openweathermap.

org/data/2.5/weather?q = ’ + str(js[‘city’]) +
‘&APPID=b2a86......’)
result = r.json();
//Parse response to get temperature in Kelvin temperature =
(str(int(math.ceil((result

[‘main’][‘temp’]) - 273.15))) + u‘\u00b0’); //Parse response to

get weather conditions condition = result[‘weather’][0]
[‘description’]
 2) Section B: This section shows how time and date infor-
mation is retrieved from the computer system. A code
snippet is as follows:
//Assign current system date and time
now = datetime.datetime.now();
//format value to get current time & date
time = (now.strftime(‘%-H:%M %p’));
date = (now.strftime(‘%b %d, %Y’));
my_date = date.today();
day = (calendar.day_name[my_date.weekday()]);
This section also contains an indicator showing the status of
the voice assistant. Green means that the voice assistant is
enabled while red means that it is disabled. The voice assistant
is automatically disabled after five minutes of inactivity. Once
this happens, the user needs to be authentication again to
continue using the system.
3) Section C: This segment handles the showcase of
multi-media content (e.g., recordings, pictures, and sound
documents). This segment works with an audience that
always checks the AWS basic line for new demands. For
each new solicitation, the audience gets interactive media
documents coordinating the solicitation utilizing the Google
custom web crawler portrayed previously. Area C in Figure
10 shows a video being played. This is in light of the
accompanying solicitation "Alexa, request that the savvy
reflect play pacific edge."
4) Section D: This segment shows news features pulled
from 16 news stations. A news source is shown like
clockwork. The code extricate underneath shows how this
segment was executed.
//Declare a counter to track progress count = 0;
//Load RSS feed URLs in list
l =[‘http://rss.cnn.com/rss/cnn_topstories.rss’,
‘http://www.tmz.com/rss.xml’];
//Use feedparser to load feeds
content = [];
f = feedparser
while count < len(l):
try:
temp = f.parse(l[count])
if len(temp[‘entries’]) > 0:
content.append(temp)
count = count + 1
except:
count = count + 1
pass
//Get title/name of first RSS station title =
(content[0]‘feed’][‘title_detail’]
[‘value’]);
//Get the first feed for the first station feed = (content[0]
[‘entries’][0]
[‘title_detail’][‘value’]);
https://cse.google.com/cse to obtain an API key and custom
search engine ID. A Python snippet is shown below to
demonstrate how to query a custom search engine.
query = ‘python programming language’; sv =
build(‘customsearch’, ‘v1’,
 developerKey = key);
result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType = ‘png’,
https://cse.google.com/cse to obtain an API key and custom
search engine ID. A Python snippet is shown below to
demonstrate how to query a custom search engine.

IV. LITERATURE SURVEY

S.NO PROJECT TECHNIQUES
1. Voice Controlled Smart Mirror with Google assistant
Multifactor Authentication IoT

2. A Mobile-Programmable Smart Mirror for IoT

Ambient IoT Environments

3. Smart Interactive Mirror Display IoT

4. The Future of Smart Dressing Mirror: An open IoT

innovation concept video

5. Google Assistant Controlled Home IoT

Automation Google Assistant
and Android

6. Automation by Voice Commands IoT

7. Smart Mirror : A novel framework for IoT

interactive display

8. Smart Mirror : A time-saving and affordable IoT

assistant

9. Smart Mirror using Raspberry Pi as a security IoT

and vigilance system.

10. Smart Mirror for Smart Life IoT

V. CONCLUSION

The system would be able to deliver promising

Output i.e every appliances connected to the arduino can be
controlled through the voice command and the
mirror would be able to deliver the notifications ,
texts , to do list and opening of mobile applications
through the voice command. And even surveillance
can be done through the mirror.

VI. REFERENCES
● Voice Controlled Smart Mirror with
Multifactor Authentication, Adokiye
Charles Njaka, Na Li, and Lin Li,
Feb 2019
● A Mobile-Programmable Smart Mirror
for Ambient IoT Environments,
Mohammed Ghazal, Tara Al Hadithy,
Yasmina Al Khalil, Dec 2017
● Smart Interactive Mirror Display,
Varsha Singh & Devi Singh, Feb 2018
● The Future of Smart Dressing Mirror:
An open innovation concept video,
Guangyu Gao & Chenchen Bai,
Mar 2016
● Google Assistant Controlled Home
Automation, Manish Prakash Gupta,
May-2018