SMART INTERACTIVE MIRROR WITH VOICE ENABLED CONTROL FOR
HOME AUTOMATION USING GOOGLE ASSISTANT
Hariharan P.S [1], Arunkumar A.M [2], Jagadeesh Chandra C.M [3], Darshan M.M [4] ,Saravana Kumar [5]
[1][2][3][4]
Final Year Students Department Of CSE , DSATM
[5]
Associate Professor Department Of CSE , DSATM
Abstract— Internet of Things, Smart Home, and Smart Cities
have become intriguing issues as of late. The improvement of these
zones becomes spirally because of the rise of cutting edge shrewd
gadgets. Savvy mirrors are another expansion to the brilliant home
family that has been getting a ton of consideration these days by both
business producers and the scholarly community. This paper portrays
how a Raspberry Pi gadget can be utilized to improve such mirrors
with insight and security. The objective is to build up a financially
savvy smart mirror framework that functions as an ordinary mirror,
yet in addition have the option to show different sorts of data, for
example, climate, time and area, recent developments, and clients.
The mirror can give sight and sound administrations while
guaranteeing top of the line security over the whole framework .And
home apparatuses, for example, lights, fans can be controlled through
the voice directions utilizing Google colleague.
File Terms shrewd mirror, Raspberry Pi, counterfeit intelli-gence,
voice acknowledgment, face acknowledgment, Internet of Things,
Alexa voice administration
I. INTRODUCTION
Brilliant Home has become a rising subject as of late in both
scholarly community and industry. With the exception of the well-
created business Internet of Things (IoT) items, for example, keen
entryway locks, savvy fittings, and brilliant indoor regulators, shrewd
home has pulled in countless individuals to chip away at self-
persuaded ventures utilizing minimal effort equipment, for example,
Raspberry Pi [1]. The known Raspberry Pi based brilliant home
ventures spread an assortment of themes, from Raspberry Pi fueled
surveillance cameras to robotized pool controllers [2] [3]. Because of
its minimal effort, such undertakings are exceptionally well known in
instruction, where understudies can get occupied with hands-on
ventures
In this paper, the creators built up a shrewd mirror sys-tem. As
another expansion to the shrewd home family, brilliant mirrors
increased expanding consideration as of late. There have been some
fascinating undertakings distributed on the Internet [4]. For example,
Philips HomeLab [5] made an astute individual consideration
framework that executed an Interactive Mirror [6] to give altered
administrations to clients. The framework is equipped for showing
TV encourages, screen the most recent climate, etc. The mirror has a
LCD show joined with a reflected surface and a processor to give the
expected administrations. Another undertaking created by Sam Ewen
and Alpay Kasal at Lit Studios [7] is a touch and signal useful mirror.
The fundamental technique for connecting with the mirror is by
contacting the screen. Five understudies at Chalmers University in
Sweden interacting with the mirror is by touching the screen. Five
understudies at Chalmers University in Sweden planned the HUD
Mirror [8]. They utilized a two-path mirror to permit the LEDs
mounted behind to light up the data. The framework can show data,
for example, time, climate, temperature, and a toothbrush clock.
The savvy reflect portrayed in this paper contrasts from the
previously mentioned undertakings mostly in that it offers
exceptional highlights to improve the framework security through
biometric authentica-tion, a voice right hand with included Alexa
abilities and sight and sound capacities. The biometric confirmation
is actualized with multi-variables, facial and voice, to guarantee a
higher security level than that gave by just voice which has been
created in Amazon Alex and Google Home. Additionally, re-
verification is authorized if the client has not been collaborating with
the mirror for some time. In this way, it can shield the framework
against a few conventional security dangers, for example, session
commandeering or man-in-center assault. Also, the voice partner and
the mixed media show actualized in the mirror cause clients to
appreciate the insight of the savvy home.
.
The guide of this paper is given as pursues: Section II
depicts the review of the framework. Segment III talks about
the subtleties of the framework structure and execution.
Segment IV exhibits the exploratory examination, and an end
is given in Section V.
II. SYSTEM OVERVIEW
The savvy reflect framework is intended to be an easy to
understand arrangement that offers disentangled and customized
benefits as a mirror. Consequently, the framework was organized to
give the client an unlimited oversight over all administrations and
capacities through voice communication. Figure 1 delineates a
schematic diagram of the framework, which can be separated into
three unmistakable parts:
Physical structure
Smart mirror software
Cloud services
The physical structure of the framework comprises of a single
direction acrylic reflect put over a showcase screen, a Raspberry
Pi gadget, and a mouthpiece. The arrangement permits the
impression of light beams from one surface of the mirror while enabling light from the presentation to go through.
Weather and
Location Weather and location
API

News Headlines AuthenticationFramework RSS News Feeds

Smart Mirror

VoiceAssistant
Request
Native Skill Amazon Webservices

AWS Lambda
Custom Skill Custom Webservices
Host

Response

Processed Response to Custom Skill Request

All the above three figures represent the already existing model.

1st figure represents the embedding working of a smart mirror assemble

The keen mirror programming was created utilizing Python
programming language. It comprises of a few modules which
handle singular elements of the framework, including climate
data, current city, time and date data, news sources and sight and
sound presentation. The mirror shows white characters over a
On the off chance that a face is recognized in a picture
dark foundation for expanded perceivability from the opposite
side of the mirror. outline, the casing is spared. This progression is rehashed
until 20 photos of the client are spared and entitled with a one
The physical structure of the framework comprises of a single of a kind whole number and the picture check (going 1 - 20).
direction acrylic reflect put over a presentation screen, a A one of a kind whole number is produced for every client in
Raspberry Pi gadget, and a mouthpiece. The arrangement permits
the framework. At that point the preparation procedure
the impression of light beams from one surface of the mirror
while enabling light from the showcase to go through. So a client begins. All the spared pictures are stacked into an exhibit,
can see his refection notwithstanding the substance appeared on named faces, and went into a recognizer. The recognizer is
the presentation. The arrangement is at long last encased with a made as an occasion of the OpenCV Local Binary Patterns
wooden casing to give the regular mirror feel to the framework. Histograms class. Neighborhood Binary Patterns (LBP)
depicts a strategy for extricating valuable highlights from face
III. SYSTEM DESIGN AND pictures. Fundamentally, this technique condenses the nearby
IMPLEMENTATION
structure in a picture by contrasting every pixel and its
This area subtleties how the framework is structured and neighbors.
devel-oped. Initial, two key segments of the framework,
//Create a LBH face recognizer object recognizer =
verification system and voice-based direction, are exhibited. cv2.createLBPHFaceRecognizer() //Train images and extract
At that point the whole keen mirror framework is portrayed. face features //Each user has a unique integer Id
Figure 3 shows the structure of the system.in an image. Face recognizer.train(faces, np.array(Id))) //Save extracted features
preparing is the way toward examining a few photos of a to face database recognizer.save
similar individual to become familiar with the highlights
explicit to the person. This procedure makes a database of the
known clients. Face forecast is the way toward distinguishing
a known or pre-enlisted individual in another picture. With the
help of OpenCV, two modules, to be specific client enrollment
and client confirmation, were created in the verification
system. Figure 4 shows the general methodology of face
acknowledgment and enlistment.

User Registration - face detection and training

This module was actualized by taking pictures with a

Raspberry camera. The pictures are passed into an OpenCV
course classifier to distinguish human countenances. A falling
clas-sifier is a model that is pre-prepared with a few hundred
face pictures, called positive models, and negative models -
discretionary pictures of a similar size. An example code bit is
appeared as pursues:
//Import opencv Library
import cv2
//Loading required cascading classifiers
face = cv2.CascadeClassifier(‘frontalface.xml’) //Load image and
convert to grayscale mode
img = cv2.imread(‘image.jpg’)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) //Face
detection in converted image
faces = face.detectMultiScale(gray, 1.3, 5) //Save image
containing faces to disc cv2.imwrite(‘image’+ str(Id))
A. Voice Based Command Wake Word Detection
The framework underpins communication among client and Like other voice-controlled frameworks, this venture actualizes
framework. Clients can control all capacities utilizing voice the utilization of a wake word. A wake word is a word or expression
directions, includ-ing enrollment, confirmation, looking, and marking that is articulated by the client to inform the framework that he is
out. The usage highlights three essential advancements: Google's prepared to give a guidance. The, endless supply of the wake word,
starts to sit tight for guidelines. The keen mirror utilizes the
"Discourse to-Text [12], Google's "Custom Search Engine [13], and
expression "shrewd mirror" as the wake work. When the shrewd
Amazon's "Alexa Voice Service (AVS)" [14].
mirror recognizes the wake word, it tunes in for the accompanying that are promptly accessible to clients of Alexa empowered gadgets.
guidance, for example, login, logout and so on. Maybe the most amazing ability of AVS is that it permits the
Pass phrase Detection formation of custom aptitudes that can be utilized to improve the
extravagance of client experience. The quantity of existing custom
Pass phrase detection refers to accepting and validating a abilities runs into the many thousands
voiceprint from the user. This happens before user enrollment and
user authentication. This task made another Alexa ability called "Shrewd mirror". This
name speaks to what the utilizations state to show that they need to
Registration Module collaborate with the expertise. The expertise is made by visiting
https://developer.amazon.com/alexa-abilities unit and choosing to
The enlistment module works somewhat unique in relation to the make another Alexa aptitude and posting the aims for the new
two cases depicted previously. This module gives the client a brief expertise. Plans are the different reasons/aims a client could have for
and sits tight for a reaction. The module initially requests that the conjuring the expertise. When the goals have been determined, the
client choose and state his username. It at that point tunes in to the clients need to make a cooperation model. A collaboration model
username before mentioning a book translation from the Speech-to- speaks to how clients communicate with the new ability. The model
Text API. The client is provoked to affirm if the content translation is contains different proclamations a client may state to cooperate with
precise after which the framework sits tight for a reaction, either
every one of the goals in the new expertise.
"yes" or "no". Once more, a book translation of the clients reaction is
gotten. On the off chance that it is "yes", the framework continues to
When the model has been fabricated, the Alexa gadget (e.g., the
finish the enrollment. On the off chance that it is "no", the framework
brilliant mirror) associated with the amazon engineer record would
comes back to the initial step to gather username.
have the option to get to the aptitude. An ability named "keen mirror"
for example may contain a plan called "getTime" and an example
Google Custom Search Engine: it is a free stage that empowers
articulation like "disclose to me the time". In the event that a client
designers to perform modified pursuits dependent on Google Search.
says "Alexa, request that shrewd mirror disclose to me the time", the
This administration likewise enables clients to penetrate down to just
Alexa Voice Service would perceive that the client is conjuring the
those genuine site pages that are pertinent. With a functioning
brilliant mirror ability.
Google account, clients can make a free custom web crawler by
visiting the Google custom hunt
Https://cse.google.com/cse to get an API key and custom internet
searcher ID. A Python piece is told underneath to show the best way
to question a custom web crawler.
query = ‘python programming language’; sv =
build(‘customsearch’, ‘v1’,
developerKey = key);
result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType =
‘png’,’).execute();

The above code scrap makes an exhibit called "result" which

contains the URLs to five pictures of PNG type that match the
depiction of Python programming language. This undertaking
utilized a custom web search tool to get sight and sound substance
dependent on client voice search inquiries, which is talked about in
detail in the following segment

Amazon Alexa Voice Service (AVS): AVS is a plat-structure that

Amazon has given to permit engineers incorporate "Alexa" into their
tasks. With an AVS empowered item, clients can stream media, get
refreshes on climate, traffic, and news, just as ask general
information inquiries. Alexa (the cerebrum behind the AVS
framework) is activated when a client says the wake word, "Alexa".
When the gadget identifies the wake word, it starts to tune in for a
solicitation. The solicitation is then handed-off to cloud based voice
administration. It is along these lines a necessity to have a working
Internet association. The cloud-based voice administration deciphers
the solicitation and sends an order back to the gadget or for this
situation, the savvy reflect framework.

Extra/custom abilities - Skills are directions that Alexa can

process. They are basically applications however custom-made
explicitly for AVS. Alexa gives a wide scope of custom aptitudes
 This section also contains an indicator showing the status of
the voice assistant. Green means that the voice assistant is
enabled while red means that it is disabled. The voice assistant
is automatically disabled after five minutes of inactivity. Once
this happens, the user needs to be authentication again to
continue using the system.
3) Section C: This segment handles the showcase of
multi-media content (e.g., recordings, pictures, and sound
documents). This segment works with an audience that
always checks the AWS basic line for new demands. For
each new solicitation, the audience gets interactive media
documents coordinating the solicitation utilizing the Google
custom web crawler portrayed previously. Area C in Figure
10 shows a video being played. This is in light of the
accompanying solicitation "Alexa, request that the savvy
Smart Mirror Software Display
reflect play pacific edge."

4) Section D: This segment shows news features pulled

B. Smart Mirror Software from 16 news stations. A news source is shown like
clockwork. The code extricate underneath shows how this
The shrewd mirror programming is created utilizing python
segment was executed.
master gramming language and PyQt 4 [17]. PyQt is a Python
authoritative of the cross-stage GUI toolbox Qt, executed as a //Declare a counter to track progress count = 0;
Python module. The savvy reflect has a presentation that is //Load RSS feed URLs in list
comprised of four unmistakable segments as enclosed Figure 10 l =[‘http://rss.cnn.com/rss/cnn_topstories.rss’,
‘http://www.tmz.com/rss.xml’];
1) Section A: This section displays weather and location //Use feedparser to load feeds
information. The data is obtained from two APIs, namely content = [];
f = feedparser
freegeoip [18] and openweathermap [19]. Below is a code while count < len(l):
snippet showing how to extract data using the APIs. try:
temp = f.parse(l[count])
//Call API to fetch city and region name
if len(temp[‘entries’]) > 0:
url = ‘http://freegeoip.net/json/’ + ip; r = requests.get(url); content.append(temp)
js = r.json(); count = count + 1
except:
//Parse response to obtain city and region response =
count = count + 1
(str(js[‘city’]) + ‘,’ +
pass
js[‘region_name’]);
//Get title/name of first RSS station title = (content[0]‘feed’]
[‘title_detail’]
//Call API to fetch temperature and weather conditions
[‘value’]);
//Data retrieval using developer ID (APPID)
//Get the first feed for the first station feed = (content[0]
r= requests.get(‘http://api.openweathermap.
[‘entries’][0]
org/data/2.5/weather?q = ’ + str(js[‘city’]) + [‘title_detail’][‘value’]);
‘&APPID=b2a86......’)
result = r.json();
//Parse response to get temperature in Kelvin temperature =
(str(int(math.ceil((result https://cse.google.com/cse to obtain an API key and custom
[‘main’][‘temp’]) - 273.15))) + u‘\u00b0’); //Parse response to
get weather conditions condition = result[‘weather’][0][‘description’] search engine ID. A Python snippet is shown below to
demonstrate how to query a custom search engine.
query = ‘python programming language’; sv =
2) Section B: This section shows how time and date infor- build(‘customsearch’, ‘v1’,
mation is retrieved from the computer system. A code snippet developerKey = key);
is as follows: result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType = ‘png’,
//Assign current system date and time
now = datetime.datetime.now();
//format value to get current time & date
time = (now.strftime(‘%-H:%M %p’)); https://cse.google.com/cse to obtain an API key and custom
date = (now.strftime(‘%b %d, %Y’)); search engine ID. A Python snippet is shown below to
my_date = date.today();
day = (calendar.day_name[my_date.weekday()]);
demonstrate how to query a custom search engine.

IV. LITERATURE SURVEY S.NO PROJECT

1. Voice Controlled Smart Mirror with Google assistant Allows the user to The system is not
Multifactor Authentication IoT control home designed for
appliances using interaction 6.
at Automation
smart mirror display deeper levels
through voice
2. A Mobile-Programmable Smart Mirror for IoT Allows the users to The system is not
Ambient IoT Environments create and manage capable of 7. Smart Mirro
their profiles based controlling all the interactive d
on what they wish to activities through
display on the smart mobile
interactive mirror
3. Smart Interactive Mirror Display IoT Reduces time needed User would8.not Smart Mirro
in a user’s daily able to access the assistant
routine and provides full features of
a merger of user and smart mirror
technology
4. The Future of Smart Dressing Mirror: An open IoT Solves serious They do not
innovation concept video difficulties to make provide quick
9. Smart Mirro
decision on which access to and vigilanc
clothes to wear or information about
match their clothes the products in the
every time, every store, available
day models and10.sizes Smart Mirro
5. Google Assistant Controlled Home IoT Proposes a cost Accurate results
Automation Google Assistant effective voice are not provided
and Android controlled (Google by the assistant
Assistant) home
automation
V. CONCLUSION  Voice Controlled Smart Mirror with
Multifactor Authentication, Adokiye
Charles Njaka, Na Li, and Lin Li,
Feb 2019

 A Mobile-Programmable Smart Mirror

for Ambient IoT Environments,
Mohammed Ghazal, Tara Al Hadithy,
Yasmina Al Khalil, Dec 2017
 Smart Interactive Mirror Display,
Varsha Singh & Devi Singh, Feb 2018
 The Future of Smart Dressing Mirror:
An open innovation concept video,
Guangyu Gao & Chenchen Bai,
Mar 2016
 Google Assistant Controlled Home
Automation, Manish Prakash Gupta,
May-2018
The system would be able to deliver promising
Output i.e every appliances connected to the
arduino can be controlled through the voice
command and the mirror would be able to deliver
the notifications , texts , to do list and opening of
mobile applications through the voice command.
And even surveillance can be done through the
mirror.

VI. REFERENCES