See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326613331

LIDAR Assist Spatial Sensing for the Visually Impaired and Performance
Analysis

Article  in  IEEE Transactions on Neural Systems and Rehabilitation Engineering · July 2018

DOI: 10.1109/TNSRE.2018.2859800

CITATIONS READS
9 940

8 authors, including:

Carolyn Ton Abdelmalak Omar

Pennsylvania State University Pennsylvania State University
2 PUBLICATIONS   9 CITATIONS    2 PUBLICATIONS   9 CITATIONS   

SEE PROFILE SEE PROFILE

Michael J Bernstein Yi Yang

Pennsylvania State University Pennsylvania State University
80 PUBLICATIONS   3,076 CITATIONS    18 PUBLICATIONS   37 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

LIDAR Assist Spatial Sensing (LASS) for the Visually Impaired and Performance Analysis View project

Many Labs project View project

All content following this page was uploaded by Yi Yang on 01 November 2018.

The user has requested enhancement of the downloaded file.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering
LIDAR Assist Spatial Sensing for the Visually
Impaired and Performance Analysis
Carolyn Ton+, Abdelmalak Omar+, Vitaliy Szedenko, Viet Hung Tran, Alina Aftab, Fabiana Perla,
Michael J. Bernstein and Yi Yang*
Abstract— Echolocation enables people with impaired or no
vision to comprehend the surrounding spatial information
through reflected sound. However, this technique often requires
substantial training and the accuracy of echolocation is subject to
various conditions. Furthermore, the individuals who practice this
sensing method must simultaneously generate the sound and
process the received audio information. This paper proposes and
evaluates a proof of concept Light Detection and Ranging
(LIDAR) Assist Spatial Sensing (LASS) system, which intends to
overcome these restrictions by obtaining the spatial information of
the user’s surroundings through a LIDAR sensor and translating
the spatial information into stereo sound of various pitches. The
stereo sound of relative pitch represents the information regarding
objects’ angular orientation and horizontal distance, respectively,
thus granting visually impaired users an enhanced spatial
perception of his or her surrounding areas and potential obstacles.
This study is divided into two phases: Phase I is to engineer the
hardware and software of the LASS system； Phase II focuses on
system efficacy study. The study, approved by the Penn State
Institutional Review Board (IRB), included 18 student volunteers
who were recruited through the Penn State Department of
Psychology Subject Pool. This study demonstrates that
blindfolded individuals equipped with the LASS system are able
to quantitatively identify surrounding obstacles, differentiate their
relative distance, and distinguish the angular location of multiple
objects with minimal training.
Index Terms— blind, Laser radar, navigation aid,
neuroplasticity, sound scape, echolocation.
I. INTRODUCTION
A S of 2010, the World Health Organization estimates that
approximately 285 million people worldwide are afflicted
by visual disabilities ranging from moderate impairment to
complete vision loss [1]. The most commonly used mobility aid
for these individuals is the white cane, a 1.0-1.5 meter rod that
can extend the user’s reach to perceive obstacles in two
dimensions through tactile feedback. Through tactile feedback,
the white cane provides adequate information for the user to
achieve basic mobility. However, it lacks the ability to detect
obstacles above a user’s waist (i.e., tree branches or signs) or
beyond the reach of the cane, approximately two steps
ahead[2][3]. Computer-vision-enhanced white cane such as the
Manuscript received XX, XXXX; revised XX, XXXX; accepted XX,
XXXX. Date of publication XXXX, 20XX; date of current version XX ,XX
XXXX. The project is funded by Penn State University at Abington ACURA
and startup fund. * Corresponding author: Yi Yang, email:yzy1@psu.edu.
Color version of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital object indentifierXXXXX
1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Smart Cane improves white cane’s function by integrating a 3D
camera and object recognition algorithm. The system verbally
warns the user when preregistered obstacles such as stairs are
detected [4]. In addition to adding new functions to the white
cane, other research on solutions to enable sensory-substitution
have been carried out.
In the absence of one sense, cross-modal neuroplasticity
enables the brain to adapt to the loss by re-directing activity to
other areas [5][6]. The current research has explored a wide
range of technology-mediated sensory-substitution to
compensate for vision loss.
The BrainPort V100 is an electronic vision aid that translates
digital information collected from a video camera to tactile
stimulation interpreted through an electrode array placed on the
user’s tongue [7]. The tongue functions as an analog of the
retina [8], while the electrode array generates patterns of
electro-stimulation to depict images from a camera, thus
enabling the user to interpret an object’s shape, size, location,
and motion.
Other devices have emphasized on auditory-spatial
interpretation since the auditory sensory channel is the highest
in bandwidth following vision. One such auditory-visual
substitution device converts digital images into simplified line
drawings and generates a musical score that can be recognized
as optical images [9]. The vOICe relies on a similar procedure
by converting images captured by a video camera to greyscale
and acoustically send the “images” as soundscapes through
headphones. The soundscapes are digital scans of the user’s
surroundings that represent an image’s vertical axis with
frequency and relates brightness with sound volume [10]. Haigh
et al. investigated the efficacy of the vOIce device by prompting
users to identify the orientations of a Snellen E and
demonstrated a promising level of acuity in participants with
minimal training [11]. Another work on substituting vision with
audio used head-related transfer functions (HRTF) to generate
auditory signals that provides information about the orientation
of the users’ head relative to geomagnetic North [12]. Although
promising results have been demonstrated, the previous
auditory-visual substitution devices are unable to collect and
convey the obstacles’ distance information around the user.
Other distance detecting electronic travel aids, such as
Miniguild, K-sonar, and iGlasses, rely on ultrasound sensors to
detect obstacles and send warnings to the users through either
sound or tactile feedback. Nonetheless, these devices can only
detect obstacles that they are pointed to, and lacks in resolution
due to the technology [13].
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering
Mammals such as bats and dolphins, including humans, have
developed natural sensory-substitution capability such as
echolocation. Previous studies have shown that humans with
impaired vision are able to achieve a high degree of
independence and self-reliance once they learn echolocation
[14]. Further studies using MRI and PET reveal that
echolocation activates both cortical auditory and visual area of
the human brain, where the latter is not known for auditory
processing [14][15][16]. Similar to bats and toothed whales, the
person utilizing echolocation has to be both the sound source
and receiver. However, both bats and toothed whales retain
advanced sensory motors to correctly process and interpret
vocal and audio signals and dynamically adapt to the received
sound[15]. As for humans, one study has shown that blind
individuals with higher computation in the occipital cortex
achieve higher accuracy in sound localization[17]. Many other
factors influence the accuracy of echolocation for humans, such
as training, obstacle distance, signal duration, binaural hearing,
as well as others [18]. Another study has shown that both head
and body motion contribute to optimal echo-acoustic
orientation detection, which requires the person be active in
both signal generation and receiving stages [19][20].
Fig. 1. LASS system operation flowchart.
Most evidence suggests that human experts in echolocation
are unable to achieve the precision level of other mammals that
specializes in echolocation because human anatomy is not
evolved to generate ultrasonic sound, which is critical in
achieving high resolution and directional echolocation [21].
Biologically inspired artificial echolocation devices have been
described to overcome these imitations. One such proposed
device, Sonic Eye, utilizes a specialized speaker to generate
1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
chirped ultrasonic sound modeled after bats’ calls [22] to
achieve high spatial resolution. After receiving the returning
ultrasound sound signals, the device plays the sound at a slower
speed to enable humans to hear the information. Users interpret
the spatial information by comparing the time difference
between the output sound and its echoes [21]. Another research
has focused on recording a large number of mouth-clicks
generated by human echolocation experts to study the beam
pattern and spectra-temporal descriptions, which enables the
development of synthetic echolocation sound [23].
In this paper, the authors propose a proof of concept LASS
system that emulates echolocation. Through utilizing a Light
Detection and Ranging (LIDAR) device, a computer and a
stereo headphone set, the LASS system collects the surrounding
spatial data of the user and translates them into soundscape
information. Compared to previous devices that utilized
ultrasound as the signal source [21], LIDAR is adopted as the
detection source due to its shorter wavelength and focused
beam, which translate into to higher spatial resolution compared
to ultrasound. Another advantage is the ability to scan the area
without the user’s head movements [21][24]–[27]. The LIDAR
collects data by emitting bursts of near-infrared light scanning
from right to left in the user’s frontal area. The LASS system
sends the collected spatial data to a computer for processing.
Similar to the solutions proposed in [21] [22], we applied head-
related transfer function (HRTF) to represent the lateral
information of objects. The angular data collected by the LASS
system is translated into stereo sound to represent spatial
orientation. There is not sufficient study to characterize
human’s ability to gauge distance using pulse-echo delays [28].
Therefore, instead of generating synthetic echo effects, the
LASS system translates the distance information into frequency
related sound signals that can be optimized to the users’
reaction: closer objects are denoted by higher frequencies
relative to further objects. The LIDAR is mounted on a chest
harness worn by the user so it can turn with the user and collect
spatial information right in front of the user.
Compared to natural echolocation, the LASS system delivers
several advantages. First, the LASS system generates and
receives the signal so that the user can focus on interoperating
the spatial information. Second, no head or body movements
are required from the user to achieve optimal performance since
LIDAR constantly scans the area in front of the user. Third, the
system requires much less training time and can be dynamically
optimized for each user. As shown in the result section, users
with minimum training are able to achieve promising
performance.
II. LASS SYSTEM
The system operation flowchart is shown in Fig. 1. Starting at
0 degree, the LIDAR collects spatial information in front of the
user; a computer receives the spatial information from the
LIDAR, and the system delivers the spatial information to the
user through stereo sound headphones. This process repeats
itself for scanning angles between 0 degrees to 180 degrees.
A Hokuyo URG-04LX Scanning Laser Rangefinder is used
as the main detection device. The LIDAR emits an infrared
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
laser beam at a wavelength of 785 nm. The Laser falls under
class 1 laser according to IEC which is defined as safe under all
conditions of normal use [29]. A rotating motor controls the
angular scanning range of the LIDAR with a maximum range
of 240 degrees, and the angular resolution is 0.00614 rad, or
0.3518o. The maximum detection distance of this LIDAR is
4.1m, with a ±2% accuracy. The LIDAR device weights 170
gram, with a size dimension of 50×50×70 mm [30]. The LIDAR
collects two-dimensional spatial information within a
horizontal plane relative to the user; data collected are
subsequently saved in the format of [Angle, Distance]. The
LASS system integrates the LIDAR via a chest harness that
mounts the device in front of the user. Therefore, the LASS
system scans the spatial information at the user’s chest level.
During operation, the user can change the area of interest by
turning one’s chest. The spatial data collected in front of the
user are then sent to the computer for processing.
A flexible cable connects the LIDAR device to a computer.
The spatial information within the angle of 180 degrees in front
of the user is gathered and processed; the angular data are
translated into stereo sound and distance data into audio signals
at different frequencies accordingly. To assist users in better
discerning relative distances [31], the system’s maximum audio
frequency is set to 650 Hz which is within the range of peak
sensitivity for humans.
The stereo sound from the headphones gives the user the
directional information of the obstacles. Humans are able to
localize a sound source by identifying the intensity and time
difference between the left and right ear [32]. As shown in
Fig.2, assuming that an obstacle generates sound from a signal
point, then the travel distance for the sound to reach the left ear
is d1, and d2 for the right ear. Since d1 > d2, it means that the
sound will arrive at the right ear ∆t earlier than the left ear,
where
∆t = (d1 – d2) / speed of sound
The intensity of sound is inverse proportional to r2, where r is
the distance between the sound source and ear. Therefore, the
sound intensity at the right ear is greater than that of the left ear.
Fig. 2. Sound localization is caused by both time and intensity difference
between the left and right ears.
In order to generate stereo sound encoded with this angular
information through the headphones, we utilized a built-in
function from SoundStream class in SFML library [33], an open
1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Transactions on Neural Systems and Rehabilitation Engineering
source library for developing multimedia applications. The
angular information of the obstacle is transformed into intensity
and time difference between the left and right ears through
HRTF [17][23][32], enabling the user to have a directional
perception of the obstacles through sound. The stereo
headphone set, Audio Technica AUD ATHAD500X, is used in
the LASS system. It has an Open-Air design for confront,
furthermore, the design enables the user to hear ambient sound.
The LASS system inversely maps distances to audio
frequencies. It generates a higher pitch for closer obstacles than
further obstacles. A nonlinear weight is applied to the distance
to frequency mapping so that closer obstacles will have a more
drastically raise in pitch. Through testing, the team used the
flowing parabolic equation to translate distance information
into audio frequency:
𝑓 = 𝑎𝐷2 + 𝑏𝐷 + 𝑐 (2)
where 𝑓 is the frequency of the sound; D is the distance of the
obstacle; a, b, and c are constants. The maximum frequency
fmax is set to 650Hz for a minimum distance of 0 meters; the
minimum frequency fmin is set to 170Hz for a maximum distance
of 6 meters, and the medium frequency fM of 490Hz is set for a
distance of 2 meters. The constants a, b, and c are solved by
substituting (650, 0), (490, 2) and (170, 6) into equation (2).
This audio frequency range is chosen to ensure that the spatial
distance information can be easily differentiated by the user
without causing much discomfort [34].
When in operation, the system scans a semicircle area in front
of the user every 13 seconds. Fig.3 shows a sample of scanned
spatial information. The green area indicates processed spatial
data, and the red area indicates unprocessed spatial data.
Obstacle 1 and Obstacle 2 indicate two sample obstacles at
different locations. During operation, the user hears the sound
of Obstacle 2 first, coming from the 2 o’clock direction. Then
the user hears the sound of Obstacle 1 coming from the 10
(1) o’clock direction. Obstacle 2 has a higher pitch than that of
Obstacle 1 because it is closer to the user.
Fig. 3. LASS system scan sample.
III. PROCEDURE
A total of 18 adult participants successfully finished the
system efficacy on human subjects. All reported normal vision
and hearing, and had no experience with the LASS system prior
to the study. The choice of normal vision participants is line
with previous proof of concept studies [4][21][35][36] that
utilized the audio channel for spatial recognition, and previous
studies have demonstrated similar auditory based spatial
learning results between groups of blindfolded sighted, late-
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering
blind, and congenitally blind participants [36]-[37]. The
participants were gathered through Penn State’s psychology
department subject pool where students were given credit
towards their classes if they participated in research projects
performed on campus. The procedures were approved by the
Penn State IRB, and all participants gave informed, written
consent.
Fig. 4. Participants wearing the LASS system and equipped with a laser
pointer.
The experiment was carried out at the Penn State Abington’s
indoor basketball court due to its open space and minimal sound
interference. Figure 4 shows a participant equipped with the
LASS system. The participant was blindfolded and equipped
with a laser pointer to point at perceived obstacles during the
study, and the computer is stationed behind the user to process
the data. The field setup in front of the participant is shown in
Fig.5, where the location of the obstacles in the field is
identified as (Angle, Distance). Obstacles are placed along the
lines with angles of 30o, 60o, 90o, 120o, and 150o. The distance
between the participants and the obstacles range from 2m to 4m.
The shaded area in Fig.5 illustrates the range where the
obstacles were positioned.
The obstacle comprises a project display board with a
dimension of 36 in. x 48 in. A target is placed on each obstacle
for system efficacy study. As shown in Fig. 6, each target is
divided into 5 regions, with Region 1 indicating the highest
accuracy, and Region 3 the least.
The experiment evaluates the efficacy of the LASS system
through asking the participants to carry out three tasks.
Task 1: Identify the number of obstacles in the field.
Task 2: Rank the distance of the obstacles in order.
Task 3: Use a laser pointer to point towards the obstacles. The
location of the laser spot on the target is recorded for the
accuracy analyses.
At the beginning of the experiment, a 5-minute introduction
was given to the participants on the LASS system and the
1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
purpose of this research. Then, the participants were asked to
wear the LASS system, and put on the blindfold and stereo
headphones. A 10-minute calibration exercise prior to the
experiment was carried out to ensure that all participants
understood the functionality of the devices and became
sensitized to variances in pitch and stereo sound. Following this
step, participants were asked to complete the tasks described in
the experiment without feedback regarding outcomes.
90°
120° 60°
Q L
T
M
150° N P S 30°
U
R
K
180° 0°
r=4 r=3 r=2 r=1
Fig. 5. Experiment field setup. r (m) represent the radius of the circle. The
letters represent the location of the obstacles in task 3, their coordinates are
shown in Table 2.
Fig. 6. The target on the obstacle.
A. CALIBRATION
Prior to the tasks described in the experiment, all participants
went through a 10-minute training to get familiar with the
LASS system. During this exercise, two obstacles were placed
at (30o, 4m) and (150o, 3m) respectively. The participants were
blind-folded and asked to identify the number of the obstacles
and rank their distance. The participants were able to visually
assess their results during each task, thus allowing them to make
adjustments.
The second exercise was designed for the participants to get
familiar with the laser pointer. An obstacle was placed at (90o,
2m). The blindfolded participants were given a laser pointer and
were asked to point at the obstacle according to the stereo sound
from the headphones. Verbal feedbacks were given to the
participants until they successfully hit Region 1 on the target.
Finally, two target obstacles were placed at (60o, 4m) and (120o,
2m), and the same procedures for one obstacle was reiterated.
B. Task 1
The blindfolded participants were asked to identify number of
obstacles placed in the field. In sets 1 and 2, two obstacles were
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering

presented. In sets 3 and 4, three obstacles were presented. The successful in identifying the relative distance of various
location of each obstacle is shown in Table 1. obstacles. However, there was a significant drop in the success
rate for Set 1, despite the fact that Set 1 only has two obstacles
C. Task 2
100

Success rate (%)

After the participants completed Task 1, they were asked to
rank the distance of the obstacles in each experiment set-up.
Prior to the participants’ ranking of the obstacles, the correct
number of obstacles was announced. 50
D. Task 3
Based on the spatial location described by the sound received 0
from the stereo headphones, the participants used the laser
pointer to point at the targets on the obstacles. Three regions Fig. 7. Success rate for Task 1: Set 1 = 94.4%; Set 2 =100%; Set 3=94.4%;
numbered 1, 2, and 3 on the target were used to measure the Set 4 = 94.4%
accuracy. The region where the laser spot hit was recorded for
analyses. The number of obstacles presented in Task 3

Success rate (%)

increased from one to three. As shown in Table 2 and Fig. 5,
100
sets 5 and 6 have one obstacle; sets 7 and 8 have two obstacles,
and sets 9, 10 and 11 present three obstacles.
50
TABLE I
SETS USED FOR TASK 1 AND 2

Set # Objects Angle (o ) Distance (m) 0

Obs. A 90 2
Set 1 Fig. 8. Success rate for Task 2: Set 1 = 61.1%; Set 2 =94.4%; Set 3=77.7%;
Obs. B 120 2 Set 4 = 83.8%
Obs. C 60 4 TABLE II
Set 2 SETS USED FOR TASK 3
Obs. D 150 3
Obs. E 30 2.5 Set # 1 Object Angle (o) Distance (m)
Set 3 Obs. F 90 3.5 Set 5 Obs. K 30 2
Obs. G 150 2
Obs. H 60 2 2 Objects Angle (o) Distance (m)
Set 4 Obs. I 120 3
Obs. J 150 4 Obs. K 30 2
Set 6
There are two obstacles in sets 1 and 2, and three obstacles in sets 3 and 4. Obs. L 60 4
Obs. M 60 3
Set 7
IV. RESULTS AND ANALYSIS Obs. N 150 4
Two obstacles were presented in Set 1. The location of each
obstacle is shown in Table 1. As for Task 1, 94.4% of the 3 Objects Angle (o) Distance (m)
participants were able to identify the number of obstacles in Set
1, where the two obstacles were separated by 30o, and 100% of Obs. M 60 3
the participants successfully identified the number of the
Set 8 Obs. P 90 2
obstacles in Set 2, where the degree of separation between the
two obstacles was 90o. For sets 3 and 4, 94.4% of the Obs. Q 120 4
participants were able to identify the number of obstacles in Obs. K 30 2
Task 1. As shown in the Fig. 7, in every set, at least 90% of
Set 9 Obs. P 90 2
participants succeeded in Task 1. The angle of separation
between obstacles did not significantly alter the success rate, Obs. R 150 3
nor did the distance of the obstacles. Obs. S 30 4
As for Task 2, 61.1% of the participants were able to correctly Obs. T 90 3
Set 10
rank relative distance between the two obstacles in Set 1,
whereas in Set 2, the success rate was 94.4%. In Set 3, 77.7% Obs. U 120 2
of the participants successfully ranked the obstacles, compared There are one obstacle in Set 5; two obstacles in Set 6 and 7; three obstacles
to 83.8% in Set 4. In sets 2, 3, and 4, the participants were rather in Set 8, 9, and 10.

1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering

in the field with closer range than other sets. The research team In Task 3, the participants used a laser pointer to point at
believes that it is due to the fact that in Set 1, both obstacles are the targets on the objects based on the spatial information
located at the same distance. When the participants were provided by the LASS system’s stereo sound and pitch. As
prompt to rank the distance of the two objects, they might have shown in Table 2, one obstacle was placed at (30o, 2m) in Set 5.
the tendency of not ranking them as the same distance. The The participants had an overall hit rate of 66.7% using the laser
results of all four sets in Task 2 are shown in Fig. 8. Overall, pointer, and 33.3% of the hits fell under region 2.
the results from tasks 1 and 2 demonstrate the LASS system’s In Set 6, one additional object was placed in the field at
ability to warn the user for potential obstacles and to identify (60o, 4m). As shown in Fig. 9, the overall hit rates were 72.2%
obstacles relative distance. and 66.7% for Obs. K and L respectively. Despite the fact that
100 two objects were placed in Set 6, the participants achieved
Set 5 Set 6 Set 6 similar hit rates as they are in Set 5. The overall and regional hit
Obs. K Obs. K Obs. L
Hit rate (%)

rates for Obs. K in Set 6 do not vary significantly compared to

Set 5. Although the overall hit rate for Obs. L is similar to that
50 of the Obs. K, there was no Region 1 hits for Obs. L. Most hits
on Obs. L appeared in Region 3 with a hit rate of 44.4%. This
is mainly due to the fact that Obs. L was 2 meters further away
0 from the participants than that of Obs. K. The results in Fig. 9
1 2 3 1 2 3 1 2 3
indicate that the LASS system indeed improves the subjects’
Fig. 9. Hit rate for Set 5, Obs. K: Region 1 = 16.7%, Region 2 = 33.3%,
ability to form a perception of relative location of one’s
Region 3 = 16.7%; Set 6, Obs. K: Region 1 = 22.2%, Region 2 = 22.2%,
Region 3 = 27.8%; Set 6, Obs. L: Region 1 = 0%, Region 2 = 22.2%, Region surroundings.
3 = 44.4%
100
100 Set 10 Set 10 Set 10
Set 7
Set 7
Hit rate (%) Obs. S Obs. T Obs. U
Obs. N
Hit rate (%)

Obs. M
50
50

0 0
1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Fig. 10. Hit rate for Set 7, Obs. M: Region 1 = 16.7%, Region 2 = 27.8%, Fig. 13. Hit rate for Set 10, Obs. S: Region 1 = 0%, Region 2 = 27.8%,
Region 3 = 16.7%; Set 7, Obs. N: Region 1 = 33.3%, Region 2 = 11.1%, Region Region 3 = 11.1%; Set 10, Obs. T: Region 1 = 11.1%, Region 2 = 27.8%,
3 = 16.7%; Region 3 = 38.9%; Set 9, Obs. U: Region 1 = 5.6%, Region 2 = 38.9%, Region
3 = 16.7%;
100
Set 8 Set 8 Set 8 Set 9 Set 10 Set 9 Set 10 Set 10 Set 8
100
Hit rate (%)

Obs. M Obs. P Obs. Q Obs. K Obs. S Obs. P Obs. T Obs. U Obs. Q

Hit rate (%)

50
50

0
1 2 3 1 2 3 1 2 3
0
Fig. 11. Hit rate for Set 8, Obs. M: Region 1 = 5.6%, Region 2 = 22.2%, 2m 4m 2m 3m 2m 4m
Region 3 = 16.7%; Set 8, Obs. P: Region 1 = 16.7%, Region 2 = 55.6%, Region
Fig. 14. Overall hit rate: Obs. K = 66.7%, Obs. S = 38.9%, Obs. P =83.3%,
3 = 16.7%; Set 8, Obs. Q: Region 1 = 5.6%, Region 2 = 22.2%, Region 3 =
Obs. T = 77.8%, Obs. U = 61.1%, Obs. Q = 50%.
22.2%;

100 Set 9 Set 10 Set 9 Set 10 Set 10 Set 8

Set 9 Set 9 Set 9
100 Obs. K Obs. S
Hit rate (%)

Obs. P Obs. T Obs. U Obs. Q

Obs. K Obs. P Obs. R
Hit rate (%)

50
50

0
1 2 3 1 2 3 1 2 3 0
Fig. 12. Hit rate for Set 9, Obs. K: Region 1 = 16.7%, Region 2 = 27.8%, 11 22 33 11 22 33 11 22 33
Region 3 = 22.2%; Set 9, Obs. P: Region 1 = 22.2%, Region 2 = 38.9%, Region Fig. 15. Region hit rate comparison: Obs. K vs. Obs. S; Obs. P vs. Obs. T; Obs.
3 = 22.2%; Set 9, Obs. R: Region 1 = 16.7%, Region 2 = 5.6%, Region 3 = U vs. Obs. Q.
38.9%;

1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering
A similar study was carried out to compare the hit rates when
2 and 3 objects were placed in the field. Their locations are
indicated in Table 2 as Set 7 and Set 8. The overall hit rate for
both Obs. M and Obs. N in Set 7 remain at 61.1%. As for Set 8,
when three objects were placed in the field, the overall hit rates
observed for Obs. M, Obs. P and Obs. Q are 44.4%, 88.9% and
50% respectively. The hit rates dropped for Obs. M and Q; this
is especially significant for the hit rates in Region 1. The
research team believes that this is mainly due to the fact that the
angle of separation between objects has dropped from 90o in Set
7 to 30o in Set 8. As for Obs. P in Set 8, when the object
appeared at 90o in front of the participant, the hit rate increased
to 88.9%, and 55.6% of the hits fell into Region 2. Therefore,
the LASS system performs well when the objects are located
right in front of the participants. As the angle of separation
between the objects decreases, the accuracy of locating these
objects decreases accordingly.
In Set 9, the team increased the separation angle between the
objects from 30o in Set 8 to 60o. A shown in Fig. 12, the overall
hit rate for Obs. K is 66.7%, which is consistent with the hit rate
occurred in Set 5 and Set 6. The overall hit rate for Obs. P is
83.3%, which is similar to the hit rate occurred in Set 8 for Obs.
P. The overall hit rate for Obs. R is 61.1%. The overall hit rates
in Set 9 are on average higher than that of Set 8, which is
consistent with the team’s previous conclusion that the LASS
system performs better for objects with larger separation angles.
In Set 10, the team studied the relationship between the
objects’ distance and system performance. As shown in Fig. 13,
the overall hit rate for Obs. S is 38.9%, Obs. T is 77.8%, and
Obs. U is 61.1%. The hit rate comparison between two objects
is shown in Fig. 14, where the overall hit rate for objects with
identical angle but different distance are paired together. The
overall hit rate decreases as the distance increases. Fig. 14
further validates that the LASS system performs better when
the obstacles are located at a 90o angle (Obs. P and T) than at
other angles such as 30o (Obs. K and S ) and 120o (Obs. U and
Q). Fig. 15 shows the regional hit rate comparison. The data
indicates that the closer obstacles also have a higher hit rate in
Region 1.
V. DISCUSSION AND CONCLUSION
There was no feedback formally collected after participants
finished the study. However, the team identified the following
challenges: a) difficulty using the laser pointer to point at the
targets. We expected that longer training would improve the
participants’ ability to accurately point at the targets, b) a full
scan takes about 13 seconds to complete, which can be a long
wait when there is no signal of interest. The 13 second scanning
time was selected to ensure first time users had adequate time
to process the spatial sound information. Further research will
focus on how to reduce this time by dynamically identifying
potential objects. One advantage of the LASS system is the
ability to scan the area in front of the users without requiring
head movements, and information sent to the user can be
dynamically optimized.
The current LASS system collects information from a 2D
LIDAR, and only planar spatial information is presented to the
users. Results from this study and [21] show that the users were
able to identify distance/depth and lateral information with little
1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
training. The ultrasound based system in [21] showed that the
object’s elevation information is difficult to judge. The team
intends to build a 3D LASS system to carry out an elevation
study of the LIDAR based system as the 3D LIDAR becomes
more affordable. The team also acknowledges that the field
setup is very ideal, as there is no background behind the
obstacles. Therefore, the LASS system will only generate a
stereo sound when it detects an object, which makes it relative
easy for the participants to identify the number of the obstacles.
This will not be the case in the real world as the structure layout
is much more complex, which requires a more intelligent
algorithm to identify objects. The team will continue to explore
improvements to address these issues raised in this study.
In this paper, the authors present a proof of concept study on
the LASS system that is inspired by echolocation. The system
intends to provide people with impaired vision of their
surrounding spatial information through stereo sound at varying
pitch. Hardware and software engineering of the LASS system
is discussed in this paper. In order to study the system efficacy,
and to further improve the system capability, 18 students from
Penn State University at Abington participated in the system
efficacy study.
Three tasks were carried out in the study. Tasks 1 and 2
focused on the LASS system’s ability to convey a general
spatial view to the participants, and Task 3 studied the
participants’ accuracy of identifying the location of objects
against varying distance, separation angle, and number of
obstacles. The results of tasks 1 and 2 validate the intuitiveness
and effectiveness of the LASS system in identifying potential
obstacles, and assisting the user to gain distance information of
the obstacles. The results from Task 3 demonstrate that despite
the short training received by the participants, the LASS system
can assistant the user to gain a general sense of the spatial
surroundings by locating obstacles at varying positions and
numbers. It further demonstrates that the LASS system is very
effective in warning the imminent hazardous obstacles, which
are closer and directly in front of the user’s path. One issue to
be stressed is that the participants were seated and never needed
to make body movements to gain better signal return. This is a
distinct contrast compared to human echolocation, where body
movements are often needed to optimize the technique.
VI. ACKNOWLEDGMENT
The participants are recruited from the Psychological & Social
Sciences Penn State Abington Research Participation Pool.
+These authors contributed equally
REFERENCES
[1] D. Pascolini and S. P. Mariotti, “Global estimates of visual
impairment: 2010,” Br. J. Ophthalmol., vol. 96, no. 5, pp. 614–618,
2012.
[2] J.-R. Rizzo, K. Conti, T. Thomas, T. E. Hudson, R. W. Emerson,
and D. S. Kim, “A new primary mobility tool for the visually
impaired: A white cane—adaptive mobility device hybrid,” Assist.
Technol., vol. 0, no. 0, pp. 1–7.
[3] M. Bousbia-Salah, M. Bettayeb, and A. Larbi, “A Navigation Aid
for Blind People,” J. Intell. Robot. Syst., vol. 64, no. 3, pp. 387–400,
2011.
[4] C. Ye and X. Qian, “3D Object Recognition of a Robotic
Navigation Aid for the Visually Impaired,” IEEE Trans. Neural
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering

Syst. Rehabil. Eng., vol. PP, no. 99, p. 1, 2017. [25]
[5] I. Rabinowitch and J. Bai, “The foundations of cross-modal
plasticity,” Commun. Integr. Biol., vol. 9, no. 2, p. e1158378, Apr.
2016. [26]
[6] C. M. Karns, M. W. Dow, and H. J. Neville, “Altered Cross-Modal
Processing in the Primary Auditory Cortex of Congenitally Deaf
Adults: A Visual-Somatosensory fMRI Study with a Double-Flash
Illusion,” J. Neurosci., vol. 32, no. 28, pp. 9626–9638, Jul. 2012.
[7] P. Grant, L. Spencer, A. Arnoldussen, R. Hogle, A. Nau, J. Szlyk, J. [27]
Nussdorf, D. C. Fletcher, K. Gordon, and W. Seiple, “The
Functional Performance of the BrainPort V100 Device in Persons
Who Are Profoundly Blind,” J. Vis. Impair. Blind., vol. 110, no. 2, [28]
pp. 77–89, 2016.
[8] E. Sampaio, S. Maris, and P. Bach-y-Rita, “Brain plasticity: ‘visual’
acuity of blind persons via the tongue,” Brain Res., vol. 908, no. 2, [29]
pp. 204–207, Jul. 2001.
[9] J. Cronly-Dillon, K. Persaud, and R. P. F. Gregory, “The perception [30]
of visual images encoded in musical form: a study in cross-modality
information transfer,” Proc. R. Soc. London B Biol. Sci., vol. 266,
no. 1436, pp. 2427–2433, 1999. [31]
[10] P. B. L. Meijer, “An experimental system for auditory image
representations,” IEEE Trans. Biomed. Eng., vol. 39, no. 2, pp. 112– [32]
121, Feb. 1992.
[11] A. Haigh, D. J. Brown, P. Meijer, and M. J. Proulx, “How well do
you see what you hear? The acuity of visual-to-auditory sensory [33]
substitution,” Front. Psychol., vol. 4, no. June, pp. 1–13, 2013. [34]
[12] F. Schumann and J. K. O’Regan, “Sensory augmentation:
integration of an auditory compass signal into human perception of [35]
space,” vol. 7, p. 42197, Feb. 2017.
[13] GDP Research, “The Miniguide mobility aid.” [Online]. Available:
http://www.gdp-research.com.au/minig_1.htm.
[14] L. Thaler, S. R. Arnott, and M. A. Goodale, “Neural correlates of [36]
natural human echolocation in early and late blind echolocation
experts,” PLoS One, vol. 6, no. 5, 2011.
[15] V. L. Flanagin, S. Schörnich, M. Schranner, N. Hummel, L. [37]
Wallmeier, M. Wahlberg, T. Stephan, and L. Wiegrebe, “Human
Exploration of Enclosed Spaces through Echolocation,” J.
Neurosci., vol. 37, no. 6, pp. 1614–1627, 2017.
[16] P. Arno, A. G. De Volder, A. Vanlierde, M.-C. Wanet-Defalque, E.
Streel, A. Robert, S. Sanabria-Bohórquez, and C. Veraart,
“Occipital Activation by Pattern Recognition in the Early Blind
Using Auditory Substitution for Vision,” Neuroimage, vol. 13, no.
4, pp. 632–645, 2001.
[17] F. Gougoux, R. J. Zatorre, M. Lassonde, P. Voss, and F. Lepore, “A
functional neuroimaging study of sound localization: Visual cortex
activity predicts performance in early-blind individuals,” PLoS
Biol., vol. 3, no. 2, pp. 0324–0333, 2005.
[18] B. N. Schenkman and M. E. Nilsson, “Human echolocation: Blind
and sighted persons’ ability to detect sounds recorded in the at Aroa Biosurgery in New Zealand in 2017 and she has been
presence of a reflecting object,” Perception, vol. 39, no. 4, pp. 483– working with Dr. Yi Yang on ACURA research since 2015.
501, 2010.
[19] J. L. Milne, M. A. Goodale, and L. Thaler, “The role of head
movements in the discrimination of 2-D shape by blind echolocation
experts,” Attention, Perception, Psychophys., vol. 76, no. 6, pp.
1828–1837, 2014.
[20] L. Wallmeier and L. Wiegrebe, “Self-motion facilitates echo-
acoustic orientation in humans,” R. Soc. Open Sci., vol. 1, no. 3,
2014.
[21] J. Dickstein, S. Teng, B. M. Gaub, J. Sohl-Dickstein, S. Teng, B. M.
Gaub, C. C. Rodgers, C. Li, M. R. DeWeese, and N. S. Harper, “A
Device for Human Ultrasonic Echolocation,” IEEE Trans. Biomed.,
the International Association of Geomagnetism and Aeronomy
vol. 62, no. 6, pp. 1526–1534, 2015.
[22] A. R. Wheeler, K. A. Fulton, J. E. Gaudette, R. A. Simmons, I. Young Scientist Award for Excellence in 2008, and the IEEE
Matsuo, and J. A. Simmons, “Echolocating Big Brown Bats, Electromagnetic Compatibility Society Best Symposium Paper
Eptesicus fuscus, Modulate Pulse Intervals to Overcome Range Award in 2011.
Ambiguity in Cluttered Surroundings,” Front. Behav. Neurosci.,
vol. 10, no. June, pp. 1–13, 2016.
[23] L. Thaler, G. M. Reich, X. Zhang, D. Wang, G. E. Smith, Z. Tao, R.
S. A. B. R. Abdullah, M. Cherniakov, C. J. Baker, D. Kish, and M.
Antoniou, “Mouth-clicks used by blind expert human echolocators -
signal description and model based signal synthesis,” PLoS Comput.
Biol., vol. 13, no. 8, p. e1005670, 2017.
[24] T. Ifukube, T. Sasaki, and C. Peng, “A blind mobility aid modeled
after echolocation of bats,” IEEE Trans. Biomed. Eng., vol. 38, no.
5, pp. 461–465, 1991.
L. Kay, “Auditory perception of objects by blind persons, using a
bioacoustic high resolution air sonar,” J. Acoust. Soc. Am., vol. 107,
no. 6, pp. 3266–3275, May 2000.
P. Mihajlik, M. Guttermuth, K. Seres, and P. Tatai, “DSP-based
ultrasonic navigation aid for the blind,” in IMTC 2001. Proceedings
of the 18th IEEE Instrumentation and Measurement Technology
Conference. Rediscovering Measurement in the Age of Informatics
(Cat. No.01CH 37188), 2001, vol. 3, pp. 1535–1540 vol.3.
D. A. Waters and H. H. Abulula, “Using bat-modelled sonar as a
navigational tool in virtual environments,” Int. J. Hum. Comput.
Stud., vol. 65, no. 10, pp. 873–886, 2007.
H. Riquimaroux, S. J. Gaioni, and N. Suga, “Cortical Computational
Maps Control Auditory Perception,” Science (80-. )., vol. 251, no.
4993, pp. 565–568, 1991.
“Safety of laser products - Part 1: Equipment classification and
requirements,” IEC 60825-1:2014, 2014. .
R. Breda, “Experimental measurement of parameters of the spatial
scanner Hokuyo URG-04LX,” Prz. Elektrotechniczny, vol. 88, no.
5, pp. 132–135, 2012.
S. A. Gelfand, Essentials of Audiology, Fourth Edi. New York:
Thieme, 2009.
D. R. Begault and E. M. Wenzel, “Headphone Localization of
Speech Stimuli,” Proc. Hum. Factors Soc. Annu. Meet., vol. 35, no.
2, pp. 82–86, 1991.
L. Gomila, “SFML Libraries.” 2017.
M. Boduch and W. Fincher, “Standards of Human Comfort:
Relative and Absolute,” University of Texas at Austin, 2009.
B. N. Walker and J. Lindsay, “Navigation Performance With a
Virtual Auditory Display: Effects of Beacon Sound, Capture Radius,
and Practice,” Hum. Factors J. Hum. Factors Ergon. Soc., vol. 48,
no. 2, pp. 265–278, Jun. 2006.
M. Geronazzo, A. Bedin, L. Brayda, C. Campus, and F. Avanzini,
“Interactive spatial sonification for non-visual exploration of virtual
maps,” Int. J. Hum. Comput. Stud., vol. 85, pp. 4–15, Jan. 2016.
J. M. Loomis, Y. Lippa, R. L. Klatzky, and R. G. Golledge, “Spatial
Updating of Locations Specified by 3-D Sound and Spatial
Language,” J. Exp. Psychol. Learn. Mem. Cogn., vol. 28, no. 2, pp.
335–345, 2002.
Carolyn N. Ton is an undergraduate student
in the Eberly College of Science at the
Pennsylvania State University. She is pursuing
a Bachelor of Science degree in biochemistry
and molecular biology with a research interest
in regenerative medicine. Her experience
includes a research and development position
Abdelmalak M. Omar is from Libya and is
pursuing an undergraduate degree in Computer
Engineering at Penn State University. He
graduated from Abington Senior High School
where he was introduced to programming.
Abdelmalak joined the ACURA program in his
freshman year. Abdelmalak was a recipient of

1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2018.2859800, IEEE
Transactions on Neural Systems and Rehabilitation Engineering

Viet H. Tran is a sophomore currently

pursuing the B.S. degree in Computer Science
from Pennsylvania State University. His
research interest includes embedded system and
system administration. Since 2016, he has been
working as a research assistant for Dr. Yi Yang
on LIDAR Assist Spatial Sensing for the
Visually Impaired. In 2017, he received a scholarship from the
2017 Multi-Campus Research Experience for Undergraduates
program.

Vitaliy Szedenko is from Ukraine and currently studies

Computer Science at Penn State University. He started as a self-
taught programmer in high school. Vitaliy joined ACURA
program in his freshman year as a software engineer.

Alina Aftab is from Pakistan. Alina is pursuing an

undergraduate degree in Computer Engineering at Penn State
University. She graduated from Parkland High School in
Allentown. Alina joined the ACURA program in her
sophomore year in Abington.

Fabiana Perla, Ed.D., chair and associate professor for the

Department of Blindness and Low Vision Studies at Salus
University. She has presented nationally and internationally and
has written numerous papers about blindness, low vision, and
orientation and mobility.

Michael Bernstein received his PhD in

Social Psychology in 2010 from Miami
University of Ohio. Since then, he has worked
at Penn State Abington, first as an Assistant
Professor and then was promoted and tenured
at Associate Professor in 2015 in the
Psychological and Social Sciences program.
His current research focuses on the
consequences and antecedents of both acute and chronic social
ostracism.

Yi Yang received his Ph.D in Electrical

Engineering in 2013 at the Johns Hopkins
University. Since 2015, he has been an
Assistant Professor in the Department of
Science and Engineering at Penn State
Abington. His current research focuses on
LIDAR Assist Spatial Sensing and analyzing
the system performance for improvements;
applying optical coherence tomography in the field of art
conservation; advancing virtual laboratory in the field of optics
and photonics.

1534-4320 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
View publication stats