Proceedings of the International Conference on Communication and Electronics Systems (ICCES 2018)

IEEE Xplore Part Number:CFP18AWO-ART; ISBN:978-1-5386-4765-3

Supermarket Automation with Chatbot and Face

Recognition Using IoT and AI
R Angeline Priyansh Rampuriya
Tannavi Gaurav
Assistant Professor Computer science engineering
Computer science engineering
Computer science engineering SRM institute of Science and
SRM institute of Science and
SRM institute of Science and Technology
Technology
Technology Chennai,India
Chennai,India
Chennai,India priyanshrampuria@gmail.com
tanvi3209@gmail.com
angeline.r@rmp.srmuniv.ac.in

Subhadeep Dey
Computer science engineering
SRM institute of Science and
Technology
Chennai,India
Subhadeep.dey30@gmail.com

Abstract—The growing population also leads to a chatbot in the supermarket to speed things up when one
growing consumer base which increases the load and needs to buy something. This chatbot can guide the
resources to cater the needs of the day-by-day shoppers about the location of each product in the store,
increasing consumers. Just as technology is being used its cost, other products similar to it and much more.
to solve all the problems so why not this one. The idea
is to make a more convenient and advanced super
market experience, where there are no cashiers or
lines so that we can shop hassle free. This project
implements Artificial Intelligence and Internet of
Things to automate a supermarket for better
efficiency. A person just needs to walk in after
scanning their QR code through the app with their
unique ID on it. It will record the customer’s presence
in the shop. The shopping carts and baskets have
sensors on them, which can detect the product entering
or being taken out. The products are placed on shelves
which have pressure sensors to detect if any product is
picked up.These shelves are closed and they only open
with your shopping cards. When you reach the
counter, you only have swipe your card again and
money will be deducted from your account according
to your purchase and a receipt will be given. We look
forward to provide this technology to various super
market chains in the country and abroad and help
them implement it with a nominal one time
investment.

Keywords—internet of things: automation; Arduino

uno 328-P; Artificial Intelligence: Chatbot;
OpenCV: Face Recognition; Product Detection; QR
Code
I. INTRODUCTION
In this project we will be using IoT and AI to collect
data by sensing activities like when the product is
removed from the shelf, when the customer enters the
shop, how much time does the customer spend on each
aisle, etc. Through the internet of things all this data can
be shared among the devices to make their shopping
experience better. The project aims to make a store
where one can have zero human interaction and still buy
the desired products. The project proposes the use of a
Fig. 1. System design
Apart from informing the customers about the products
it can also be used to promote the other products or
upcoming deals on the product which will boost the sales
of the supermarket significantly.
This project is based on the idea of assignment of IP
addresses to the various objects or ‘things’ that are
present in a supermarket. Doing so will provide a
network connectivity so that the things or devices can
connect to each other and share real-time data. This data

978-1-5386-4765-3/18/$31.00 ©2018 IEEE 1183

 Proceedings of the International Conference on Communication and Electronics Systems (ICCES 2018)
IEEE Xplore Part Number:CFP18AWO-ART; ISBN:978-1-5386-4765-3
can be uploaded to an online cloud service and used for
further analysis and study. Making use of OpenCV can
help in eliminating many of the sensors which have been
prominently used in the previous works done by
researchers on the same concept. OpenCV will enable
the system to detect and identify the various faces of the
customers and also products which they are looking to
buy.
II. RELATED WORKS
A. Chatbot
There are chatbots available which have been created
using Python. Also, some chatbots for supermarket are
already present but they have been implemented using
Facebook Messenger. This project will feature a chatbot
created using the python library chatterbot which will
combine both the two by having a Supermarket Chatbot
implemented on Python.
B. Face Recognition
Python has been used to detect faces with the help of
numpy and OpenCV. Various machine learning algorithms
are used in to detect faces using OpenCV which is by far the
most popular library for computer vision. Every object or in
this case face is further divided into smaller bite-sized tasks
known as classifiers. There can be more than 5000 classifiers
for a particular face. Checking each classifier to detect a face
can be tedious and time consuming task so to make it
executable in real-time it uses something called cascades. The
entire face is subdivided into blocks for a quick and rough
test. This quick check determines if a detailed test needs to be
carried out or not.
C. Product Detection
The current supermarkets use barcodes or RFID tags for
classifying each product available in the supermarket. It does
not have an active role to play until it is time for billing. Every
product of the same kind contains the same barcode which
can be scanned using a barcode scanning machine.
III. MAIN SYSTEM AND OPERATING FLOW
3.1 Main System
The proposed system looks to automate the entire
experience of shopping right from the entry into the store to
the exit point. This entire experience can be categorised into
5 divisions, App installation, QR entry, Exploring,
Information and Advertising and Exit.
1. App Installation- Before entering the supermarket every
customer must install the app which will enable them to
scan the QR code while making an entry and also browse
through the various products that the supermarket has to
offer. They can also know about the prices of various
products and the products which are already sold out. It
will also be mandatory for every customer to link their
credit/debit card to the app so that they can have a hassle-
free check out after They have completed shopping for the
products.
978-1-5386-4765-3/18/$31.00 ©2018 IEEE
2. QR Entry- Every customer must scan the QR code from
his/her app which will confirm the entry of the customer in
the supermarket and also store the data related to that
customer in the database (AWS) with help of the QR scan.
3. Exploring- After the entry point each customer is free to
visit any aisle which contains shelves full of different
products. Every product will have its unique product code
which is recognisable by OpenCV cascades. These codes
will be right by the camera when the product is being
removed from the shelf and then the algorithm will be able
to detect which product it is. Another camera will also
recognise the face of the customer and associate the
products which he or she is holding to their virtual cart. In
a similar way as the customer keeps on taking the products
from various shelves These products keep on getting added
on to the virtual cart of that particular customer. Since all
the products of a similar kind will have the same product
code on them it will be easier to sort the products. For
instance, if by mistake any customer puts a different
product on the different aisle where it doesn’t belong then
a buzzer goes on.
4. Information and Advertising- The chatbot developed using
chatterbot library on Python is used to tell the customers
about the new products that are going to launch or which
have launched recently. More primarily it will work as an
assist for people who are in a rush and want to know the
whereabouts of a particular product.
5. Exit- When the customer is done shopping for the
products he can simply exit by again scanning the QR code
via the installed app and leave. A few moments after
leaving the store he/she will receive a message confirming
the deduction of amount of the total bill of the purchase.
3.2 Special Operating Procedures
1. It is mandatory for every customer to install the app
which will make them eligible to enter the store. Failing
to do so will not let them enter the store or purchase
products from the store and if someone tries to forcefully
enter the supermarket beyond the QR entry point an
alarm will go on.
2. When a customer enquires about a certain product from
the chatbot then he/she will get recommendations of
similar products or other products from the same brand.
3. Chatbot is also helpful in guiding the customer about the
best route to reach the selected product.
4. When the supermarket if out of a certain product that the
customer has selected then he/she can select the option on
the app to get a notification when the product is back in
stock.
5. When any product has crossed its expiry date then the
concerned authority can be updated about it so that it can
be replaced. This can be made possible since the product
code on every product contains details about its expiry
and as soon as someone tries to grab it from the shelf an
indicator will go on which will indicate that the particular
product is not safe to use or consume.
6. In case when any customer after looking through the
supermarket decides to not buy any product and wants to
leave then he/she just needs to follow the similar
procedure and scan the QR code near the exit point.
1184
 Proceedings of the International Conference on Communication and Electronics Systems (ICCES 2018)
IEEE Xplore Part Number:CFP18AWO-ART; ISBN:978-1-5386-4765-3
IV.MODULE DESCRIPTION
The automated supermarket project consists of modules for
QR entry/exit point, app, smart aisles and virtual cart.
1. QR Entry/Exit Point- The module simple makes use of
the QR code generated on the app installed on the user’s
smartphone and the QR code scanner at the entry or exit
point. While walking in or out a customer simply needs to
scan the QR at the entry or exit point and that will mark
his arrival or departure from the store.
2. Supermarket App- The app is crucial to the project as it
generates the QR code with the customer details and also
keeps them updated on the products and its cost or
availability of it in the store. When the product is not
available then the customers can chose an option on the
app to notify them when the product is back in stock.
3. Smart Aisle- Each aisle is designed in such a way that
whenever a customer pulls out a product from any shelf the
cameras capture it and the object is detected using
OpenCV which using Machine Learning algorithms. Each
product has a unique product code which is a series of
shapes that can be easily recognizable by the computer
vision. The product code contains details of the product
like cost, weight, expiry date, etc. This concept will also
help the supermarket authority be updated about when to
replace certain products from the shelf when it has crossed
its expiry date and is no longer safe to use or consume. An
attempt to pull out such product from the shelf will switch
on an indicator so that the consumer can be informed about
it. The system also recognizes the face of the customer and
associates the product picked by the customer to his/her
virtual cart along with details like quantity of the product
and its price.
4. Virtual Cart- Virtual cart functions in a way similar to the
real-life cart. As the customer keeps on picking certain
products and moves on to the other aisle the cart receives
updates from the system on specifications like cost and
quantity and keeps on updating the same to its cloud-based
database (AWS). When the shopping duration is over then
the sum total from this database is fetched and is deducted
from the users account.
V. SENSORS USED
For sensing the data from the surroundings in the
supermarket we make use of a few sensors.
1.Weight sensor-
FSRs (force sensitive resistor) are sensors that allow you
to detect physical pressure, squeezing and weight. They are
simple to use and low cost. The FSR has 2 layers which are
separated by a spacer. The more the more the pressure is
applied, the more of those Active Element dots touch the
semiconductor and the resistance go down. The FRS is used
in rack so as the product is pickup it detects the product and
send mail the main server.
978-1-5386-4765-3/18/$31.00 ©2018 IEEE
2. IR sensor-
An infrared sensor is an electronic device, which emits
infrared ray in order to sense the surroundings. It can also be
to measure the heat of an object as well as detects the motion
of the object. The sensors which are used just sense the
infrared radiation but they can not emit them are called
passive IR sensors . The IR sensor is used in the smart cart as
it is used to detect the product which is been kept in the cart.
3. PIR Proximity Sensor-
PIR sensors are used to sense motion, it is used to detect
whether a human has moved in or out of the sensors range.
They are very small in size, affordable and low cost , low-
power consumption , easy to use and durable. So the PIR
sensors are commonly found in appliances used in homes or
businesses. The PIR sensor are often referred to as "Passive
Infrared", "Pyroelectric". It is used smart cart to detect the
position of the customer.
VI.SYSTEM ARCHITECTURE
The customer scans the QR code and enter the
supermarket and the person scan the QR code using the AWS
(Amazon Web Services) the person profile is send to the
Inventory database. If the person is regular customer they can
use the Face recognition and the detail is send to the
Customer Database . As the customer pick up the things using
the weight sensor (FSR) the system scan which item is picked
up send the purchase status to the customer E-mail address
Fig. 1. System architecture
and generates the bill. When the proceed for checkout it
checks item purchase by the customer and generate the final
bill and the data to the purchase database. The supermarket
uses the chatbot which is made using the python though
which customer can get the detail about the product and send
the data to the Chatbot database. The supermarket also has a
Smart cart which used to track customer location and product
inside the cart which used to verify the product which is taken
from the rack is kept inside the smart cart. The customer
move checkout and generate the bill verify the product which
is linked to the purchase check. The purchase database is
connected to the AWS store analytics which share the data
with the store database.
1185
 Proceedings of the International Conference on Communication and Electronics Systems (ICCES 2018)
IEEE Xplore Part Number:CFP18AWO-ART; ISBN:978-1-5386-4765-3

Conference on Wireless Sensor Network, IET, pp. 17-

VII. LIMITATION OF THIS IMPLEMENTATION
QR Scanning: The method of entry of the customer on
QR code scanning is a huge limitation since a person might
or might not carry his or her smart-phone at all times. When
a person is not carrying a smart phone and needs to buy a
product from the store he or she will be unable to do so
because of this method. There can be another method which
must be implemented in which no external piece of
technology has to be carried by the user so that anyone can
walk in hands-free and walk out with the required products.
Face Recognition: The project uses face recognition to
detect the customer’s faces and store and maintain a database
containing details about them and the items purchased. The
process requires a huge number of cameras all over the store
which is both costly and intimidating for a customer.
20, 2010.
[9] Seung-Ho Baeg, Jae-Han Park, Jaehan Koh, Kyung-
Wook Park, Moon-Hong Baeg, “Building a Smart Home
Environment for Service Robots Based on RFID and
Sensor Networks,” International Conference on Control,
Automation and Systems 2007 Oct. 17-20, pp. 1078-
1082.
[10] Xiaolin Jia1,2, Quanyuan Feng2, Taihua Fan1, Quanshui
Lei1, “RFID Technology and Its Applications in Internet
of Things (IOT),” 2nd International conference on
Consumer Electronics, Communications and Networks
(CECNet), IEEE, pp. 1282-1285, 2012.

VIII. CONCLUSION
Such an automated system when implemented gives a
better productivity and also saves the customer’s time taken
to shop at the store. It reportedly reduces the time taken to
shop by each customer by 60% which is very low as
compared to the time taken earlier by them.

REFERENCES
The template will number citations consecutively within
brackets [1]. The sentence punctuation follows the bracket
[2]. Refer simply to the reference number, as in [3]—do not
use “Ref. [3]” or “reference [3]” except at the beginning of a
sentence: “Re

[1] Marcus St¨ander, Aristotelis Hadjakos, Niklas

Lochschmidt, Christian Klos, Bastian Renner, Max
M¨uhlh¨auser. “A Smart Kitchen Infrastructure”, 2012
IEEE International Symposium on Multimedia.
[2] Alaoui, Lina, James Deen, Stephanie Ahn, and Fei Ye.
"Kitchen Manager: Your Shopping and Food Inventory
Made Easy
[3] Farahzadi, P. Shams, J. Rezazadeh, and R.Farahbakhsh,
“Middleware technologies for cloud of things-a survey,”
DigitalCommunications and Networks, Elsevier, 2017.
[4] Zanella, N. Bui, A. Castellani, L. Vangelista, and M.
Zorzi,“Internet of things for smart cities,” IEEE Internet
of Things Journal,vol. 1, no. 1, pp. 22–32, Feb 2014.
[5] Chao Ai, Huaming.)X ³0RGHUQ Internet of things
equipment design EDVHG RQ 5),' WHFKQRORJ\´,
Application of Electronic Elements, vol.9, No. 129, pp.
75-77, (2007).
[6] 0]Lin Luo, M.N.S. Swamy, Eugene I. Plotkin, ―A
Modified PCA algorithm for face recognition‖,
Proceedings of IEEE 2003.
[7] Kirti Chawla, Gabriel Robins, and Liuyi Zhang, “Object
Localization Using RFID,” 5th International Symposium
on Wireless Pervasive Computing (ISWPC), IEEE, pp.
301-306, 2010.
[8] Rong Chen, Li Peng, Yi Qin, “Supermarket Shopping
Guide System based on Internet of things,” International

978-1-5386-4765-3/18/$31.00 ©2018 IEEE 1186