You are on page 1of 6

2014 13th International Conference on Control, Automation, Robotics & Vision

Marina Bay Sands, Singapore, 10-12th December 2014 (ICARCV 2014) Fr47.5

Overview of Assistive Technologies for


the Blind: Navigation and Shopping
Karen Duarte José Cecı́lio Pedro Furtado
Department of Informatics Engineering Department of Informatics Engineering Department of Informatics Engineering
University of Coimbra, Portugal University of Coimbra, Portugal University of Coimbra, Portugal
ksduarte@student.fisica.uc.pt jcecilio@dei.uc.pt pnf@dei.uc.pt

Abstract—One of the major driving forces behind technology II. U SED T ECHNOLOGIES
development is the establishment of equality among everyone.
It is intended that the strategic use of technology complements Radio Frequency Identification, RFID, is a method of
competencies or diminished capacities, improving self confidence
and autonomy. Blind and partially sighted people often deal with
automatic identification by radio signal. A tag identifies and
this inequality in the performance of tasks or daily activities. object and is able to respond to radio signals from radio
The presented work reviews some of the proposed technologies waves transceivers. Tags can be active or passive: active tags
to assist visually impaired people in common activities, specifically have external power supply and therefore a greater range.
navigation and going shopping. The Electronic Product Code, EPC, is a universal identifier
designed to be stored on an RFID tag.
I. I NTRODUCTION UPC, Universal Product Code, is the foundation of EPC.
UPC is a barcode symbology that stores information within
Visually impaired people are defined as those with little black bars and white spaces. There are several UPC types that
or no visual capacity, also named blind and partially sighted can be used for many different applications. Quick Response,
people. These conditions often limit peoples ability to perform QR, are 2D barcodes codes that can easily be deciphered by a
common tasks and affects their quality of life. It is estimated camera. Those codes usually store text information, numbers,
that there are over 285 million[23] people with this disability geographic coordinates or other information.
worldwide. Bluetooth is a wireless transfer technology of a small range
Among the activities affected by visual impairment, navi- widely used on mobile phones, computers and accessories.
gation plays a fundamental role, since it enables the person to ZigBee is another wireless communication technology used
independently move in safety. The heterogeneous environment, between devices, known by its low powered operation and long
easily perceived by visually enabled people, is hardly known distance range. Unlike Bluetooth and ZigBee, that are personal
by partially sighted people. The environment features are area networks, WiFi is a local area network with wider range
perceived through hearing and touch, usually blind people use and enables devices to exchange data or connect to the Internet
dogs and white canes for that. using radio waves.
Another challenging task for these people is independent The famous GPS standing for Global Positioning System,
shopping. In most supermarkets information about products is a satellite based positioning system, enabling to specify
is only presented for visually enabled people, making blind the position of any point on the earth’s surface with an
people dependent on others. unobstructed line of sight to four or more GPS satellites.
The work here presented provides an overview over some The Global System for Mobile Communications, GSM, is the
technologies intended to support visually impaired people in standard used by mobile phones. GSM operates in the 900MHz
some daily tasks, such as navigation and going shopping. or 1800MHz for the second generation networks, 2G.
Systems and applications here presented were divided into Infrared radiation is part of the electromagnetic radiation
three categories: Assistive Technologies for Navigation, As- with longer wavelengths than visible light, 700 nanometers.
sistive Technologies for Shopping and Assistive Technologies Infrared radiation is widely used on remote controls of televi-
for Navigation and Shopping. Within shopping activity it is sions and other devices. Ultrasounds are sound waves with a
included knowing and finding product categories or specific frequency greater than the human hearing range, approximately
products. 20kHz. Ultrasounds are often used in medical imaging and on
The paper is organized as follows: Section II introduces the boats to calculate distances.
major used technologies in the systems, Section III presents Voice recognition software is used to convert speech to
systems for assisted navigation, Section IV presents systems words. Several software can be used for this purpose, the
for assisted shopping, Section V systems for both navigation choice of the best to use depends on the application features.
and shopping assistance and Section VI concludes the paper. Speech synthesizers execute the opposite operation, converting
from words and sentences to speech.

978-1-4799-5199-4/14/$31.00 ©2014 IEEE 1929


III. A SSISTIVE T ECHNOLOGIES FOR NAVIGATION D. BlindAid
A. RFID System 1 RFID based system is composed of a portable RFID reader
in communication with a portable computing device, such
The environment must be covered with RFID tags and thus as smartphone[13]. Users location and route to a destination
the objects intended to be found, allowing the user to travel are calculated based on prepared map data. An effective tags
through the space and to find those objects[20]. Each tag placement enables user localization everywhere and direction
is documented on a database and has a corresponding voice of travel calculation in order to plan route and deliver the
recording or text data that will guide the user. The user device right instructions to the user. Path calculation is performed
is an RFID reader and a earphone, the information travels via by Dijkstra’s shortest path algorithm.
middle-ware. RFID reader collects the tags on the vicinity and User inputs may be received through voice instructions or
sends their identity code to the remote computer which will using buttons on the device. The system works only where
process the information and send the voice instructions to the RFID tags are placed, so it is necessary to install a lot of
user, earphone will deliver it. The database is implemented on RFID tags. System relies on user skills to detect and avoid
the server and can easily be modified, making each update obstacles, but enables the user to use is own smartphone, a
available to all users. The remote computer, earphone and device he already knows how to work with.
RFID reader must be in permanent wireless communication.
This proposal ensures user positioning and guidance, but
relies on user abilities to detect obstacles. The user is en-
abled to indicate a desired destination or product, among the
documented by RFID tags, and be lead to it. A drawback of
this proposal is that the environment and products have to be
labeled with RFID tags.

B. RFID System 2
S. Chumkamon proposes a system with three subsystems:
the track infrastructure, the navigation device and the navi-
gation server[3]. The first one is related to the RFID tags
network, it intends to place tags on a tactile orientation system
on the footpath, which is used by visually impaired people for
orientation. The navigation device is composed of an RFID
reader, a microprocessor, a communication module, a user Fig. 1: BlindAid[13].
interface and a memory module.
When the user wants to start the navigation, the commu-
nication module of the navigation device asks the server, via E. Smart-Robot
GPRS network, for a route and stores the information received. Smart-Robot is a robot proposed for indoor and outdoor
This way, communication with the server is only required on navigation[24]. Indoor navigation is performed using RFID
the route start or when the user wants to start a new one, or technology and outdoor by GPS. The robot is equipped with an
gets lost. The system enables user guidance at least in the analog compass for orientation, ultrasonic and infrared sensors
decision points, but obstacle detection relies on user’s abilities for obstacle avoidance, a speaker and a vibrating glove for
and sensory skills. instructions delivery, and a keypad for user inputs.
Destination is recorded by the keypad and the robot per-
C. RG forms navigation checking for obstacles on the way. Indoors
Robotic Guide, RG, is a robot that escorts the user through the Smart-Robot travels close to the wall where it finds best
a route, it is intended to assist users in navigating unknown RFID signal intensity. Although it is known that the GPS
environments[9]. Indoors navigation is performed by potential system has an error of some meters, this system has the strong
fields and by finding empty spaces around the robot. Potential advantage of performing indoor and outdoor navigation.
fields is an algorithm that calculates robot motion by attracting
it to the desired end and repulsing it from obstacles. This F. GPS System 1
technique enables the robot to follow corridors without ori- This system integrates a mobile phone with camera and GPS
entation sensors. RFID tags are used for navigation and for technology, and motion and magnetic induction equipment[15].
robot location. The aim is to provide information to travel a specific route
This system has the advantage of not requiring a global calculated for the system regarding user desired destination,
map of the environment to navigate, the last position is always this will comprise traffic and obstacle information.
stored on the connectivity graph and precise locations are Localization and route calculations are based on GPS,
known when the robot receives an RFID tag. and orientation and motion are received by the motion and

1930
to deliver right alarms: fast and slow moving obstacles through
sound and vibrations for different levels of warning.
The presented system only helps the user on his arbitrary
path, not enabling the blind user to travel through a new path
with confidence, in order to not get lost. The user is enabled
to use the system on any environment, receiving information
about obstacles everywhere.

I. Mobile Phone System


The system presented here intends to provide information
about public transportation usually available visually through
posters or informative signs, and to provide users orientation
inside a building or public space[5]. The proposal is available
anywhere through a simple voice call from any mobile device.
Orientation and navigation is performed by a discrete and
non-obstructive system of acoustic emitters that with user
interaction is able to know users position and to calculate
Fig. 2: Smart-Robot.[24]. a route to the desired point of interest. Navigation will be
processed by the emission of sounds by alternated emitters and
regular user positioning, so the environment must be provided
magnetic induction equipment. Obstacle information will be with several sound emitters. A strong advantage is the usage
deteted through mobile phone camera. of user’s own mobile phone.
Although it is known that GPS technology does not deliver
precise positioning information, as told before, the system as J. Vision Based System
the advantage of enabling the visually impaired user to use his User device are acrylate protective glasses holding two
own mobile phone, avoiding the need to learn how to use a cameras that transmit video streaming to the operator via Wi-
new one. Fi[22]. Operator sees the context of environment 3D so he can
estimate distance of obstacles. With a joystick the operator
G. GPS System 2 handles the choice of the audio sample heard by the user in
A similar system has a built in navigation map used for the memory of a portable computer he is carrying. Instructions
route calculations, using a weighted graph. The GPS system is to user’s guidance are artificially generated simulating virtual
used for user positioning and frequent actualization of the route source of an audio signal. A drawback of the presented system
and instructions to deliver to the user. The navigation device is is the frame rate and size of images, the achieved was 15
composed of a power supply module, a mobile communication frames per second and 640x480 pixels.
and positioning module, a voice module, a processor module Since the system is based on stereoscopic vision, there
and a blind people input keyboard[2]. are no environment adjustments, but position can only be
As the unique location and navigation system used is GPS, ambiguously known as it is based on users vicinity images.
the proposed system relies on user skills to detect obstacles. A heavy drawback is that it always has to have an operator
available.
H. Ultrasounds System
This proposal uses ultrasonic technology to detect whether
are obstacles in front of user and if they are moving or
may be dangerous. The user devices embodies the informa-
tion processing equipment, a micro-controller, an ultrasonic
transmitter and receiver and the alarm apparatus, sound and
vibration. The micro-controller is simultaneously in connection
with the information equipment and the alarm apparatus,
thus the information equipment is connected to the ultrasonic
device[19].
The ultrasonic transmitter emits ultrasonic waves that are
reflected by the environment, and thus obstacles, and the
ultrasonic receiver receives the reflected waves. Then the
information processing equipment processes the signal and
calculates obstacle distances and motion information (moving Fig. 3: Prototype proposed[22].
or stationary, and moving slow or fast). Micro-controller has

1931
IV. A SSISTIVE T ECHNOLOGIES FOR S HOPPING As the previous systems, the presented here is designed
A. Voice Shopping Guide only for product identification, so the user will not receive
instructions to reach it. The system relies on user’s smartphone,
This system places voice emitters near the products on the
but considering the products are not yet equipped with QR
shelves that displays information when the user approaches
code and Braille seals, there are several modifications needed
them[6]. Infrared sensors are used to detect the proximity of
to me done on the environment. A great advantage of this
blind users, that proximity distance ranges since 10cm to 50cm.
system is that the user is enabled to receive information about
Information displayed can introduce the section the user just
the unique product he is holding.
entered, or identify product types in the user’s vicinity, and can
be easily changed anytime to adapt the environment changes. E. RFID and EPC Based System
he required changes on the environment will depend on the An EPC-RFID combined system was proposed by Sreekar
level of specification intended to achieve, the user does not Krishna[7]. A RFID reader collects the EPC code from the
need to carry any device to use the proposed system. Although products on the shelves as user approaches them, and the
the system can provide much information, the user will rely product’s information is accessed by user’s PDA or smartphone
on his skills to have sure he has the specific wanted product. and delivered to the user via synthesized speech. The users
B. GroZi device communicates with the central server in the supermarket
through Internet. Central server is essentially a relational
A different proposal is the presented by GroZi[14][18] database with information of all the products which performs
project. This proposal is composed of a web site for wanted the task of associating EPC sent by the portable computing de-
product’s list creation, a computer vision software for products vice to a unique product’s information. The interface between
recognition in stores and a portable device that can execute the user’s device and the database is implemented as a middle-
computer vision algorithms and give the user haptic and verbal ware layer. RFID reader is mounted on a bracelet and ensures
feedback. The user compiles a shopping list of products and Bluetooth communication with the user’s device.
uploads it on the portable device. In the store, the user wears Since the product is identified by an unique EPC, the user
a hand glove with a small camera and vibrating motors to is enabled to receive information about one specific product.
point aisles, the application will return information about With information about the surrounding products the user can
products there and instructions to find the desired product. be lead to the wanted product, but all products need to be
When a product on the shopping list is detected in the video equipped with EPC or RFID identification.
stream, the application will give the user either haptic or
verbal instructions to reach it. GroZi uses in vitro and in situ
images: the first are taken under ideal lighting and perspective
conditions, the second are obtained from video streams in the
store.
Since it relies on images, minimal environment adjustments
are required, however the system does not indicate where the
products are, only tells what is the product the user is pointing
at.
C. Barcode System
The system is based on barcode recognition: barcodes are
detected in vision range of the user. With code information
and its known location, the user is informed about commodity Fig. 4: ShopTalk[16].
regions, products arrangement, and name, trademark and price
of a single product[17]. Barcode identification comprises a set
of image processing steps, such as filtering to noise and texture F. Trinetra
detection. Trinetra is a solution based on RFID and UPC
Assuming the system can be implemented in a simple identification[11]. Trinetra is composed of an Internet and
device, or even in the smartphone, this system as the advantage Bluetooth enabled cell phone, text-to-speech software and a
of enabling the user to user his own portable device. portable barcode reader. The shopper’s phone receives informa-
tion from the barcode reader, RFID reader or keypad; the first
D. QR Code System two are received via Bluetooth. When an ID is received, phone
This proposal is implemented on a smartphone equipped application matches it with a most recently used ID’s looking
with QR code reader software[1]. The user uses the mobile for a match. When no match is found, phone communicates
phone to scan QR codes on products, the device will connect with the server over the Internet, the server will match the ID
to Internet and a verbal description of the product will be with all product ID’s available in the store. Once a product
delivered. To enable blind people to find the QR code, its is identified, its description will be returned to the user.
location is marked with a Braille seal. Information is delivered to the user via synthesized speech.

1932
Although this solution identifies well each product, it does
not allows the user to navigate autonomously within the store
and easily find wanted products. When the environment does
not is provided with RFID tags, the UPC identification prevails
and it is considered that all products have a unique UPC.
V. A SSISTIVE T ECHNOLOGIES FOR NAVIGATION AND
S HOPPING
A. Virtual Walking Stick
This system intends to allow detection of obstacles, identifi-
cation of colors, recognition of products or objects by barcode
or RFID reading, orientation, reproduction of audio content,
signaling the user on dark environment and recognition of
letters, words or sentences through a digital camera. This
device is aimed to provide as much information as the user
may need, since date and hours information to points of
interest in public spaces, replacing the traditional white cane
by a stick with technology that collects information about
the environment without having to physically touch objects Fig. 5: RoboCart[8].
and obstacles[21]. Ultrasonic sensors are used for obstacles
and depressions detection (R4) and alarms or warnings are
delivered by vibrations and sound alerts. A digital compass the user a product selection interface, when confirmed the
and a GPS system are installed for outdoor navigation and selection guides him to the vicinity of the product. Once there,
positioning information. Product recognition is enabled by RoboCart uses the user’s egocentric frame of reference to
digital camera or RFID reader and a wireless connection find the product and a barcode reader ergonomically modified
to an existing database on the establishment, which delivers to help the alignment with the shelves, where the products
information about the product or object. barcodes are placed. The floor of the store is converted into an
RFID-enabled surface, where each RFID tag had its own 2D
B. BlindShopping coordinates. This RFID tags are used as recalibration areas,
BlindShopping is an RFID and QR-code based mobile when the robot reaches a recalibration area its localization is
solution[12], that demands inexpensive off-the-shelf technol- well known.
ogy. The products are grouped into different categories, and Two drawbacks of RoboCart are that the user has to scan
these are divided into product types which are separated into all barcodes until reach the target one, and he can completely
concrete brand products. The supermarket surface is mapped miss the barcode he is looking for and never find the product.
with cells containing shelves and passageway cells, and RFID On the other hand, the RFID identification enables constant
tags are distributed throughout the floor creating an RFID and precise positioning.
map. BlindShopping maps the IDs of the RFID tags within
a cell to navigation and product localization, instructions are D. ShopTalk
given to the user through voice messages. Once in the target ShopTalk[16] is a system composed of a processor, a
aisle, this solution offers support for product recognition by numeric keypad and a barcode scanner. ShopTalk guides the
shelf section identification, by QR code, product own ID, user to the vicinity of the target product with vocal instructions
or barcode scanning. The user will use personal smartphone with route directions and descriptions of the store layout.
and the supermarket must have a wireless network and server The topological map of the store, used for route calculations,
installed. RFID readers, attachable to the white cane, are lent is a directed graph whose nodes are decision points like
to visually impaired clients by supermarket. aisle entrances or cashier lane entrances. Route directions are
Once the floor is mapped with RFID tags, user’s position is constructed from a database of parametrized route directions,
always known, but obstacle detection relies on user abilities. and the keypad is used to set instructions as done, the system is
As the system implies the deployment of an RFID web to unaware of the shopper’s actual location and orientation. The
user’s location and QR codes on the shelves for products barcode scanner was modified to align more easily with the
identification, there are several environment required changes. shelves where the product barcodes are. The key data structure
The user is enabled to specify a wanted product and be guided that associates barcodes with aisles, aisle sides, shelf sections,
to it, but he can not specify a list of products in one time. specific shelves and relative positions on the shelves. Users
location can be known once he/she scans a barcode on an
C. RoboCart aisle, allowing the user to use the system even if he/she gets
Another solution uses a robot to help the visually impaired lost.
costumer to find wanted products. RoboCart[4] presents to The user is guided to the vicinity of the wanted product,

1933
but relies on own sensory abilities for obstacles detection. [5] Guidance, navigation and information system especially adapted
for blind or partially sighted people. https://www.google.com/patents/
The system comprises a whole new user device, however uses WO2010002284A2?cl=en Google Patents, 7 January 2010.
barcodes on the shelves that may already exist, so minimal [6] Induction voice system capable of performing shopping guide for blind
environment modifications are required. people. https://www.google.com/patents/CN202775358U?cl=en Google
Patents, 13 March 2013.
1) ShopMobile: ShopMobile is an updated version of [7] KRISHNA Sreekar et al., A Wearable Wireless RFID System for Accessi-
ShopTalk that runs on a mobile phone. A vision-based barcode ble Shopping Environments. BodyNets ’08 Proceedings of the ICST 3rd
scanning method[10] is proposed. Once again are applied international conference on Body area networks, Article No. 29.
[8] KULYUKIN Vladimir and KUTIYANAWALA Aliasgar, Accessible Shop-
stabilizers to help the visually impaired user align the camera ping Systems for Blind and Visually Impaired Individuals: Design Require-
with shelves, where product barcodes are placed. The system ments and the State of the Art. The Open Rehabilitation Journal, 2010, 3,
finds barcodes in images: when part of a barcode is detected, 158-168.
[9] KULYUKIN Vladimir et al., RFID in Robot-Assisted Indoor Navigation
instructions are given to slide the phone along the shelf in one for the Visually Impaired. Proceedings of 2004 IEEE/RSJ International
specific way. Once the target barcode is reached, the user takes Conference on Intelligent Robots and Systems. September 28 - October
a product from the shelf. 2, 2004, Sendai, Japan.
[10] KULYUKIN Vladimir and KUTIYANAWALA Aliasgar, From ShopTalk
This system has the strong advantage of enabling the user to ShopMobile: Vision-Based Barcode Scanning with Mobile Phones for
to use his own mobile phone, does not requiring effort to learn Independent Blind Grocery Shopping. RESNA Annual Conference, 26-30
how to use a new one. June 2010, Las Vegas, Nevada.
[11] LANIGAN Patrick et al., Trinetra: Assistive Technologies for Grocery
Shopping for the Blind. 10th IEEE International Symposium on Wearable
VI. C ONCLUSION Computers, 11-14 October 2006.
The technologies proposed cannot really be generally com- [12] LÓPEZ-DE-IPIÑA Diego, LORIDO Tania and LÓPEZ Unai, Indoor
Navigation and Product Recognition for Blind People Assisted Shopping.
pared, as some are only intended to assist navigation and other Third International Workshop, IWAAL 2011, Held at IWANN 2011,
to find products. Among the systems for assisting navigation, Torremolinos-Mlaga, Spain, June 8-10, 2011.
GPS is widely used as it is a common technology. However it [13] MAU Sandra et al., BlindAid: An Electronic Travel Aid for the Blind. The
Robotics Institute - Carnegie Mellon University, Pittsburgh, Pennsylvania.
is already known that GPS have an error of a few meters, for May 2008.
pedestrian navigation that is a wide error. And so, for indoor [14] MERLER Michele, GALLEGUILLOS Carolina and BELONGIE Serge,
navigation, which means more accuracy, RFID technology is Recognizing Groceries in situ using in vitro Training Data. SLAM,
Minneapolis, MN 2007.
more used. For system intended to find specific products, a [15] Navigation mobile phone suitable for blind people. https://www.google.
relevant technology used is barcode and QR code identifica- com/patents/CN101980515A?cl=en Google Patents, 23 February 2011.
tion, as it is already used for products unique identification it [16] NICHOLSON John, KULYUKIN Vladimir and COSTER Daniel,
ShopTalk: Independent Blind Shopping Through Verbal Route Directions
is highly advantageous to support systems on it. and Barcode Scans. The Open Rehabilitation Journal, 2009, 2, 11-23.
This study presents some proposal systems and applications [17] Shopping guide method and apparatus for blind in supermarket based on
designed for support daily tasks of blind and visually impaired bar code identification. https://www.google.com/patents/CN101474020B?
cl=en Google Patents, 8 January 2010.
people, with relevance to navigation and going to the supermar- [18] TRAN Robert et al., GroZi: Grocery Shopping Assistant for the Blind.
ket. In order to integrate blind people on society, more public Continuity Report, ENG 100L.
spaces should be equipped with such systems or alternative [19] Ultrassonic technology based electronic terminal for blind peo-
ple and navigation method thereof. https://www.google.com/patents/
methods with the same intention. From the proposed systems CN102743271A?cl=en Google Patents, 24 October 2012.
here presented it is clear that RFID is a promising technology [20] VARPE Kanchan and WANKHADE M. P., Survey of Visually Impaired
in this field, due to its clear accuracy and low acquisition cost Assistive System. International Journal of Engineering and Innovative
Technology (IJEIT), Volume 2, Issue 11, May 2013.
and deployment. [21] Virtual walking stick for assisting blind people. https://www.google.com/
patents/WO2011104589A1?cl=en Google Patents, 1 September 2011.
ACKNOWLEDGMENT [22] VTEK Stanislav et al., New Possibilities for Blind People Navigation.
International Conference on Applied Electronics (AE), Czech Republic,
This work was partially financed by iCIS Intelli- 7-8 September 2011.
gent Computing in the Internet Services (CENTRO-07- [23] Visual impairment and blindness. http://www.who.int/mediacentre/
factsheets/fs282/en/ World Health Organization. March 29, 2014.
ST24FEDER002003), Portugal. [24] YELAMARTHI Kumar et al., RFID and GPS Integrated Navigation
System for the Visually Impaired. 53rd International Midwest Symposium
R EFERENCES on Circuits and Systems (MWSCAS). 1-4 August 2010.
[1] AL-KHALIFA Hend, Utilizing QR Code and Mobile Phones for Blinds
and Visually Impaired People. Computers Helping People with Special
Needs. Lecture Notes in Computer Science Volume 5105, 2008, pp 1065-
1069
[2] Blind people communication navigation device. https://www.google.com/
patents/CN201476794U?cl=en Google Patents, 19 May 2011.
[3] CHUMKAMON Sakmongkon, TUVAPHANTHAPHIPHAT Peranitti and
KEERATIWINTAKORN Phongsak, A Blind Naviagtion System Using
RFID for Indoor Environments. 5th International Conference on Electrical
Engineering/Electronics, Computer, Telecommunications and Information
Technology. ECTI-CON 2008. 14-17 May 2008.
[4] GHARPURE Chaitanya and KULYUKIN Vladimir, Robot-assited shop-
ping for the blind: issues in spatial cognition and product selection. Intel
Serv Robotics, 2008, 1:237-251.

1934

You might also like