Grasping Objects With A Sensory Substitution Glove

Journal Pre-proof
Grasping Objects with a Sensory Substitution Glove
Carlos de Paz , Jorge Ibáñez-Gijón , David Travieso ,

David M. Jacobs
PII: S1071-5819(22)00181-1
DOI: https://doi.org/10.1016/j.ijhcs.2022.102963
Reference: YIJHC 102963
To appear in: International Journal of Human-Computer Studies
Received date: 15 February 2022

Revised date: 5 October 2022
Accepted date: 6 November 2022
Please cite this article as: Carlos de Paz , Jorge Ibáñez-Gijón , David Travieso , David M. Jacobs ,
Grasping Objects with a Sensory Substitution Glove, International Journal of Human-Computer Studies
(2022), doi: https://doi.org/10.1016/j.ijhcs.2022.102963
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.
© 2022 Published by Elsevier Ltd.

Grasping Objects with a Sensory Substitution Glove
Carlos de Paz, Jorge Ibáñez-Gijón, David Travieso, David M. Jacobs
Facultad de Psicología, Universidad Autónoma de Madrid, 28049, Madrid, Spain
Running Head: Sensory Substitution and Grasping
Author Note
Correspondence concerning this article should be addressed to David Travieso, Facultad de
Psicología, Universidad Autónoma de Madrid, 28049 Madrid, Spain.
E-mail: david.travieso@uam.es
Highlights
 We present a new Sensory Substitution Glove intended to grasp objects by providing vibrotactile
information contingent on active exploration.
 Blindfolded participants were able to detect and grasp objects with different sizes and at
different locations.
 Hand movement during grasping showed distinguishable orientation, reaching, and grasping
phases.
 Objects were successfully grasped in 88% of the trials.

SENSORY SUBSTITUTION AND GRASPING 2
Abstract
People with a visual impairment often find it difficult to detect information about
distant objects they may want to grasp. To overcome this difficulty, we developed a sensory
substitution glove that facilitates non-visual grasping. The glove includes two vibration
motors, one on the index finger and one on the thumb. The motors vibrate whenever the
corresponding finger points toward the object. The vibration intensity increases when the
hand approaches the target. Three experiments were performed with the glove, with
blindfolded participants. Experiment 1 tested the ability of participants to point to cylindrical
targets. The absolute angular error (1.38º) was lower than the angular size of the object
(3.82º). In Experiment 2, participants aimed to grasp differently sized cylinders at different
distances. They were successful in 83% of the trials. The observed movements showed
distinguishable reaching and grasping phases. Experiment 3 manipulated the direction, size,
and distance of the targets, hence representing a more real-world situation. In this case, the
action was completed successfully in 93% of the trials. An orientation phase preceded the
reaching and grasping components. Oscillatory explorations were observed in all
experiments, permitting the detection of the information that is needed to successfully
complete the action.
Keywords: grasping, reach-to-grasp movement, sensory substitution, visual impairment,
visuomotor channels model

1. Introduction
Grasping is an essential sensorimotor function that allows humans (and several other
animals) to perform actions such as throwing, cutting, or drinking. In most cases, grasping
behavior is controlled visually (Jeannerod, 1984, 1986, 1999; Jeannerod et al., 1995; O’Shea
& Redmond, 2021; Smeets et al., 2019). That is, based on information provided by vision, the
arm and hand are moved toward the target in ways that depend on the object's position as
well as on properties such as size, orientation, and hardness. The visual control of grasping is
a sophisticated and highly-developed skill for most of us (Marteniuk et al., 1990). Visual
control of the action is not possible, however, under conditions of low visibility or for
individuals with a visual impairment.
The world health organization (WHO) estimates that more than 250 million people
around the world have a moderate or severe visual impairment (Bourne et al., 2017). Usually,
individuals with a visual impairment do not have critical manipulation problems after having
grasped an object (Castiello et al., 1993; Jakobson & Goodale, 1991; Jeannerod, 1984;
Goodale et al., 1986; Wing et al., 1986). In contrast, they find it difficult to locate the object
in their peripersonal space (i.e., the space immediately surrounding the body; Rizzolatti et al.,
1997). Compensatory strategies mitigate this problem to some extent. People with visual
impairment may, for example, explore the space around them with their hands using circular
or segmental movements, or try to keep objects at the same place.
Notwithstanding the variety of compensatory strategies, developing aids for the
localization of objects that are difficult to locate through direct contact is of utmost interest.
In this article, we present a novel vibrotactile glove and three experiments concerning the use
of this aid. The glove provides information about the distance and direction of objects, which
can be used for the real-time control of grasping. We hope that our research contributes to the
development of aids for people with a visual impairment. In addition, the analysis of non-
visual grasping behavior may be informative about the different components of similar
actions under more natural conditions.
The vibrotactile glove that we present belongs to the more general category of sensory
substitution devices (SSDs). These devices make use of one perceptual system (in our case,
touch) to transmit information that is typically detected with another perceptual system (in
our case, vision). The first well-known SSDs were developed in the sixties (Bach-y-Rita et
al., 1969; Starkiewicz & Kuliszewski, 1963). Since then, the overall purpose of SSDs has
evolved from trying to substitute vision as a whole (general-purpose devices) to trying to
replace specific visual functions (special-purpose devices; Loomis et al., 2012; Spence,
2014). Examples of special-purpose devices are the TSIGHT (Cancar et al., 2013; Lobo et al.,
2018) and the Enactive Torch (Froese et al., 2012; Froese & Ortiz-Garin, 2020; Lobo et al.,
2019). These aids use vibration to provide information about the distance toward the nearest
surface in a certain direction (the shorter the distance, the more intense the vibration).
Exploration with such devices permits users to establish perception-action couplings, or
sensorimotor contingencies (O’Regan & Noë, 2001), and hence to detect information that is
useful for navigation and collision avoidance. Following the same reasoning: When faced
with a grasping task, what would be the most relevant information to transmit with an SSD?
The mainstream model of grasping, the visuomotor channels model, holds that grasping
is one of the two main components of prehension, and that each component has its own
visuomotor channel (Jeannerod, 1984, 1999; Stival et al., 2019). The first component, the
reach, is responsible for transporting the hand toward the object. In the original formulation
of the model, this component was claimed to depend on information about extrinsic
properties of the object (e.g., position and orientation). The second component, the grasp,
concerns the opening and closing of the hand until the fingers are placed on the surface of the
object. Initially, this component was claimed to rely on information about intrinsic properties
of the object, such as size or fragility. Given that the grasp is typically initiated early during
the reach, the two components tend to unfold largely in parallel. The maximum distance
between the fingers occurs at about 60% of the total movement time (Wallace & Weeks,
1988), although this percentage has been shown to depend on experimental conditions
(Marteniuk et al., 1990). Research that followed the original formulation of the model has in
many cases demonstrated relations between the two visuomotor channels (Castiello et al.,
1993; Chieffi & Gentilucci, 1993; Jakobson & Goodale, 1991; Paulignan et al., 1991a,
1991b; Wing & Fraser, 1983).
An alternative model is the double-pointing model (Smeets & Brenner, 1999). This
model proposes that “grasping is nothing more than pointing with the thumb and [index]
fingers toward selected positions on the surface of the object” (Smeets & Brenner, 1999, p.
2). In other words, the thumb and index finger are controlled independently, trying to reach
positions on the object surface that allow lifting. According to this model, prehension does
not comprise separately controlled reach and grasp components, and the distinction between
intrinsic and extrinsic properties is not crucial to the movement control. In particular,
properties such as the hand aperture emerge as a mere consequence of the independently
controlled fingers. Smeets and Brenner (1999) demonstrated that their model can be used to
fit trajectories observed in previous studies (cf. Verheij et al., 2012). More recently, Smeets et
al. (2019) reviewed an extensive body of evidence that favors the theoretical view that the
trajectories of the thumb and index finger are controlled independently.
Some authors have argued, however, that the double-pointing model implies substantial
simplifications of actual behavior, and that the trajectory fitting that is achieved with the
model does not entail an understanding of the functional constraints that are relevant to
prehension (Newell & Cesari, 1999). Furthermore, van de Kamp and Zaal (2007) tested the
independence of the trajectories of the thumb and index finger experimentally by perturbing
the width of the target only on one side. On several occasions, van de Kamp and Zaal
observed modifications of the movement trajectories of the fingers whose object side was not
manipulated (cf. Zaal & Bongers, 2014), hence questioning at least some of the assumptions
that underlie the double-pointing model.
Relevant to our purpose, the role of real-time visual information during reaching and
grasping has also been addressed (Castiello et al., 1993; Jakobson & Goodale, 1991; Wing et
al., 1986). It has been argued that the main effect of real-time visual information is an
increase in accuracy, without qualitative changes in the pattern of prehension. For example,
in Castiello et al. (1993), blind participants who performed repeated prehension movements
showed similar overall movement patterns as blindfolded and full-vision participants, even
though they needed more time and used larger hand apertures (e.g., applied larger safety
margins). Relatedly, Bozzacchi et al. (2018) observed that the absence of real-time visual
information has an effect during early stages of the movement (altering the hand trajectories)
as well as during later parts (in some conditions leading to larger hand apertures).
In addition to the study of visually-controlled grasping, assistive devices have been
designed that provide vibrotactile information of locations in peripersonal space. Among
these devices are the PalmSight (Yu et al., 2016) and the FingerSight (Satpute et al., 2019).
These respective devices are attached to the palm of the hand and the index finger, leaving
the natural sensibility of the finger pads unaltered. The devices provide information to guide
the user toward an object, indicating a to-be-performed action (i.e., a movement direction).
As such, the devices do not rely as much on the capacity of users to explore and actively
detect information as several other devices (Díaz et al., 2012; Lobo et al., 2018; Lenay &
Steiner, 2010). Rather than emphasizing differences, however, we would like to indicate that
our research is aligned with the research concerning the PalmSight and FingerSight. As do
the authors of that research, we believe that too few SSDs have focused on the localization of
objects in peripersonal space.
As a final note, we should mention that the graspability of an object can be conceived
as a body-scaled affordance (Warren 1984; Warren & Whang, 1987). Affordances are
opportunities for action that an environment offers to a specific organism (Gibson, 1979).
The perception of affordances with SSDs has proven to be a useful concept, for example for
the climbability of stairs (Lobo et al., 2014; Travieso et al., 2015), passability of apertures (de
Paz et al., 2019; Favela et al., 2018; Kolarik et al., 2014), or collision avoidance (Jicol et al.,
2020; Kolarik et al., 2017; Lobo et al., 2019). In the present case, an object is graspable with
one hand if its size does not exceed a critical point relative to the size of the hand (Newell et
al., 1989; van der Kamp et al., 1998). For example, a handball is graspable with one hand
whereas a basketball requires two hands (for most people). Therefore, our device should
allow users (a) to move the hand toward the object on the basis of information about the
distance and direction of the object relative to the current position of the hand and (b) to
perform an appropriate hand aperture on the basis of information about the size of the object
relative to the size of the hand.
To summarize, a vibrotactile glove that is meant to facilitate grasping should ideally
allow the detection of the distance, size, and direction of objects (Jeannerod, 1984) as well as
the location of the to-be-grasped edges of the objects (Smeets & Brenner, 1999). To meet
these criteria, the glove that we developed includes a vibrotactile motor on the back of the
index finger and another one on the back of the thumb. These motors vibrate as a function of
the distance to the nearest surface in the pointing direction of the associated fingers. The
device thereby allows the detection of the information that is needed to fulfill the task,
although active exploration of the user is needed for this (Lenay et al., 1997). Apart from the
thin fabric of the glove itself, the SSD leaves the inner side of the hand free for the user to
make a precision grip with the thumb and fingertips as well as a grip with the whole hand,
and it does not perturb exploration. In addition, our aid fulfills the performance criteria
proposed by Kaczmarek et al. (1991), including maximal stimulation comfort and minimal
sensory adaptation.
We present results from three experiments. Experiment 1 was designed to test if our
glove allows blindfolded users to point to objects in their peripersonal space. Participants
pointed with the index finger, rotating the hand without displacing it. Experiment 2 used a
simple grasping task in which participants were asked to grasp objects in a fixed direction but
with different sizes and located at different distances. We expected to observe the classic
pattern of grasping under visual control, which is to say, a reach-to-grasp movement in which
the onset of the hand opening and the maximum hand aperture were adjusted to the distance
and size of the object. The third experiment was a combination of the first two experiments.
Users were asked to locate and grasp objects with differences in distance, size, and direction.
2. General Method
2.1 Ethics Statement
The local ethics committee approved the experimental protocol (UAM-CEI-94-1717)
and participants signed informed consent forms before participating in the experiment.
2.2 Participants
All participants were students at the Faculty of Psychology of the Universidad
Autónoma de Madrid (UAM). They received course credits for their participation. None of
them had participated in previous sensory substitution experiments and none of them
participated in more than one experiment of the present series. Because a right-hand glove
was used, only right-handed participants were recruited, meaning that all participants
performed the task with their dominant hand. We did not take gender into account in the
recruitment process. Even so, the percentage of female participants was much higher in all
experiments. This is so because the participants were students at the Faculty of Psychology.
In more technical areas, gender biases tend to be the opposite (Merino et al., 2020). We did
not observe noteworthy differences in performance between female and male participants.
The a priori sample size was calculated for each experiment using the G-power program
(Faul et al., 2007).
2.3 Apparatus
The experimental set-up included a chair and a wooden table (140 width × 80 depth ×
75 cm height). A chinrest was used to avoid head and trunk movements and to make sure that
all participants performed the task while sitting at the same position. From that position, the
task that was used in each of the three experiments could be performed comfortably with
movements of the arm, hand, and fingers. The table was covered with a black cloth in order
to remove light reflections that interfere with the infrared motion-capture (MOCAP) system
(Qualisys AB, Göteborg, Sweden). The experimenter and the participants communicated
verbally, meaning that no headphones or earplugs could be used. However, because of the
tablecloth and a careful positioning of the objects by the experimenter, no acoustic cues about
the target locations were available.
As to-be-grasped objects, we used plastic 10-cm-high cylinders with different
diameters. These objects were covered with adhesive tape to reduce reflections. The SSD did
not include actual distance sensors. Distance was measured by locating objects and fingers
through the MOCAP system. However, as participants did not interact with the object in
Experiment 1, we created a virtual object with the same size and position as the physical one
and computed the distance between the registered finger positions and the virtual object.
Experiments 2 and 3 were performed with the physical objects. One of the markers of the
MOCAP system was placed on top of the physical object. Trials finished automatically
whenever this marker was displaced, which is to say, whenever the object was touched, hit,
or grasped.
The SSD consisted of a thin polyester glove with two vibratory coin motors, which
were attached to the back of the second phalanx of the index finder and the thumb (Figure
1a). The motors were connected with wires to a printed circuit board (PCB). The PCB, in
turn, was connected to a digital/analog (D/A) conversion card. Six reflective markers were
used to register the position of the fingers at 120 Hz. The positions of the markers on the
glove can be seen in Figure 1. The markers themselves were registered with positional errors
lower than 1 mm by the Qualisys system. To these errors one should add some minor
fluctuations of the markers and the glove with regard to the hand. Our impression, however,
is that these errors were not relevant to the functioning of the device. Moreover, we believe
that the only effect that such imprecisions may have had on the experimental results, if any, is
a very minor increase in variability.
2.4 Functioning of SSD
We used a computer (PC Intel Core i7, 3.07 GHz) and a self-developed routine
programmed in Matlab (The MathWorks, Inc., Natick, 2016) for the online computation of
the vibration intensities of the coin motors. The program calculated if the line that crossed the
two markers on each finger intersected the perimeter of the target object using the “linecirc”
function in Matlab. If this was the case, which is to say, if the finger pointed to the object, the
coin motor on that finger vibrated (Figure 1b). The intensity of vibration was a function of
the distance to the object. In this article, we indicate the vibration intensity using the digital
voltage level (V) that Matlab sent to the D/A conversion card. This voltage level was later
adapted by the PCB to meet the specifications of the used motors (Díaz et al., 2012).
In Experiment 1, the vibration intensity was fixed at 8 V. In Experiments 2 and 3, the
vibration started at a distance of 30 cm, and the intensity increased from the minimum level
of 3.1 V at 30 cm to the maximum level of 10 V when the object was contacted. More
precisely, the intensity of vibration at distances closer than 30 cm was modulated with the
following equation:
𝑉 = 0.009 × 𝑑2 − 0.5 × 𝑑 + 10 (1)
in which distance, d, is expressed in cm. This quadratic equation was used because the
increase in vibration is more pronounced than for a linear equation at shorter distances, which
is to say, when the user is about to touch the object. The voltage levels of 3.1 and 10 V
corresponded, approximately, to frequencies of vibration of 42 and 65 Hz (Appendix A of
Díaz et al., 2012). Those levels correspond to the minimum voltage to activate the coin motor
and its maximum admitted voltage.
2.5 Procedure
The experiments included between six and eight training trials and between 25 and 80
experimental trials. Before the training trials, participants read the following instructions:
“You will sit on a chair in front of a table and perform the experiment while being
blindfolded. A cylindrical object will be placed on the table. Each trial starts from the same
initial position. Your aim is to grab the object using the SSD, without touching the object
previously. There are no time constraints and no restrictions on your movements. The SSD
has one vibrotactile motor on the index finger and another one on the thumb. If one of the
fingers points in the direction of the object, the motor on that finger vibrates. If you get closer
to the object, the vibration increases”.
After reading the instructions, participants performed the training and experimental
trials. Participants never saw the objects. Moreover, the objects were not touched until the
first training trial was performed (apart from Experiment 1, in which the objects were not
touched at all). Each experimental condition was repeated five times. The trials were
performed in a randomized order, with the exception that the same condition was never
repeated in successive trials. This exception was included to rule out that the task could be
performed to some extent on the basis of learning by mere repetition. Trials ended when
participants verbally indicated that they were pointing to the center of the target (Experiment
1) or when the target was displaced (Experiments 2 and 3).
2.6 Data Acquisition, Preprocessing, and Statistical Analysis
Hand movement data were filtered with a 4th-order low-pass Butterworth filter with a
cut-off frequency of 8 Hz. Only the horizontal dimensions of the marker positions were
considered. In our quantitative analyses, we defined the direction from the starting point of
the hand to the target location as y dimension and the orthogonal direction as x dimension.
Note, therefore, that the coordinate axes of the analyses were defined per condition, with
regard to the to-be-performed movement, and that one may expect larger movements in the y
direction than in the x direction.
We performed repeated-measures analyses of variance (ANOVAs) to test the effects of
the independent variables, after averaging data over repetitions. Greenhouser-Geisser
corrections were used whenever the sphericity assumption was violated. The level of
significance was set at p = .05. Bonferroni corrections were used for post hoc analyses. We
performed unpaired t tests in those cases in which it was interesting to test whether two levels
of an independent variable differed significantly. For variables that had one measurement for
each finger, paired t tests were used to assess differences between the fingers. Means and
standard deviations were reported for each level of the independent variables. Both data and
Matlab code are available from the corresponding author upon request.
2.7 Transparency and openness
We report how we determined our sample size, all data exclusions, all manipulations,
and all measures in the study. Both data and Matlab code are available by emailing the
corresponding author. This study was not preregistered.

3. Experiment 1: Orienting the Hand to the Object
Allowing users to detect the direction of objects is essential for a glove that is meant to
facilitate non-visual grasping. Previous studies with SSDs on the hand have demonstrated
that pointing to objects can successfully be performed with these devices. Lenay et al. (1997)
and Lenay and Steiner (2010) reported the successful detection of the position of light
sources by pointing to them. Nevertheless, the errors observed in these studies may not be
small enough to permit successful grasping. SSDs on the torso have been shown to allow a
highly accurate alignment of the body midline to objects (Faugloire & Lejeune, 2014; Lobo
et al., 2018). Lobo et al. (2018) reported mean absolute errors of 1.4º and mean signed errors
of -0.2º. Such accurate performance was achieved only under conditions with active
exploration (their Experiment 1a), hence demonstrating the crucial role of the active
exploration (de Paz et al., 2019; Lenay et al., 2003).
Pointing behavior has also been studied from the perspective of the haptic perception of
spatial relations (Fernández-Díaz & Travieso, 2011; Kaas & Van Mier, 2006; Kappers, 2018;
Kappers & Koenderink, 1999; Van Mier, 2019). In a typical experiment, blindfolded
participants are asked to orient a test bar to a sound source or to leave it parallel, collinear, or
mirroring a reference bar. The observed errors can be summarized with three main
tendencies. First, the deviations are clockwise when the test bar is located to the right of the
reference bar and counterclockwise when it is located to the left. Second, the deviations are
larger when the horizontal distance between the test bar and the reference bar is longer.
Third, the deviations are larger for orientations that do not coincide with the frontal or sagittal
plane (i.e., for oblique orientations).
Experiment 1 was designed to investigate whether our SSD allows participants to turn
their hand to targets at different locations, without displacing the hand toward the targets. No
restrictions were used with regard to the movement time and the rotational exploration. We
do not expect that the size of the errors depends on the direction of the targets, because the
continuous perception-action coupling facilitated by the glove may help users to reduce the
deviations regardless of the target direction.
3.1 Method
A total of 32 participants (30 female, 2 male) performed the experiment. Their ages
varied between 18 and 22 years (M = 19.3, SD = 0.9). Five target positions were used (Figure
2). The direction of the target ranged from 70º to 150º in steps of 20º. The object was always
located at a distance of 30 cm (from the hand position).
Given that the task involved turning but not moving to the object, participants grabbed
a fixed vertical rod as a pivot that allowed rotations but not displacements. Only the motor
and the markers on the index finger were used. The task did not imply contact with the
object. For this reason, only virtual objects were used. The object diameter was fixed at 2.0
cm, meaning that the angular size of the object as perceived from the rotation point was 3.82º.
When pointing to the object, the motor on the index finger was stimulated with a constant
driving voltage of 8 V. At the start of the actual experiment, the hand orientation was set at
110º. Trials finished when participants verbally indicated that they were pointing to the
center of the object. Subsequent trials started without returning the hand to the initial
orientation. Given that each condition was repeated five times, participants performed 5 × 5 =
25 experimental trials. Training trials were performed with target directions ranging from 80º
to 140º in steps of 20º, repeating each direction twice. The duration of the experiment was
approximately 15 min.
We calculated absolute and signed angular errors as dependent variables. The absolute
angular error was defined as the absolute value of the angular difference between the final
pointing direction of the index finger and the actual direction of the center of the target. The
signed angular error was defined as the signed value of that same difference. The value of the
signed error was negative when participants pointed to the left of the object
(counterclockwise deviation) and positive when they pointed to the right (clockwise
deviation). Further dependent variables were calculated from the exploratory movements.
These variables included the time used to perform the trials, the total angular distance
covered during the trials, and the number and amplitude of the oscillations. To identify
oscillations, we first used the function “peakdet” in Matlab to localize the local maximums
and minimums in the time-series of the x coordinate of the marker near the top of the index
finger (using a delta [threshold] of 3.0 for large oscillations and 0.5 for small oscillations). An
oscillation was then defined as a transition in the x values from a maximum to a minimum or
vice versa, and the amplitude as the difference between the maximum and minimum.
3.2 Results
3.2.1 Preliminary analysis
None of the trials were discarded due to recording problems or missing data. An
example of a trial can be seen in Figure 3. At the beginning of the trials, large movements
were typically used to discover the overall target direction. Participants then continued with
smaller movements to localize the center of the object.
3.2.2 Absolute angular errors
A repeated-measures ANOVA with the five target directions as a factor showed that
this direction did not significantly affect the absolute angular errors, F(4, 124) = 0.68, p =
.61, ηp2 = .02. This means that the size of the deviations was not significantly related to the
direction of the targets. The average angular deviation was 1.38º (SD = 0.38). The observed
deviations were substantially smaller than the angular width of the objects as computed from
the hand position (3.82º). One may therefore conclude that our SSD allowed participants to
correctly detect and point to the objects.
3.2.3 Signed angular errors

A repeated-measures ANOVA revealed a significant effect of the target direction on the
signed angular errors, F(4, 124) = 4.04, p < .01, ηp2 = .11. This effect is illustrated in Figure
4. Bonferroni post hoc tests showed that the rightmost target had a significantly (p < .05)
further leftward deviation (M = -0.83º, SD = 0.98) than the leftmost target (M = 0.04º, SD =
0.92). The averaged angular deviations per target condition were smaller than 1.0º, and so
were the standard deviations. Overall, the averaged signed error was -0.43º (SD = 0.97).
Although the signed errors were significantly different from zero, t(159) = -5.58, p < .001,
they were small if considered in comparison to the angular width of the target (3.82º). In sum,
the angular deviations were small, tended to be to the left (negative/counterclockwise), and
were less substantial for target objects that were placed further leftward.
3.2.4 Movement variables
On average, the pointing movements were completed in 5.54 s (SD = 2.99). The
movement duration was positively correlated with the absolute angular error, r(32) = .52, p <
.01; the longer the used time, the larger the error. The trial duration was not correlated with
the signed error, r(32) = -.15, p > .05. An average angular distance of 327.6º (SD = 302.6)
was covered during the trials. This angular distance was covered through oscillatory
movements. In 86.4% of the trials, participants performed at least one oscillation. Paired t
tests showed that the number and amplitude of the oscillations were larger during the first
half of the trials than during the second half (p < .01, for number; p < .001, for amplitude).
The decrease in amplitude indicates that the oscillations formed part of a process by which
participants slowly homed in on the target.
3.3 Discussion
In the present experiment, participants oriented the hand to targets. They first
determined the approximate direction of the objects with a few large oscillations. Once the
overall direction was determined, more numerous but narrower oscillations were used to
localize the center of the object. This pattern of results indicates that participants were able to
take advantage of the emerging sensorimotor contingencies. In general, when active
exploration is allowed with sensory substitution devices, performance errors are smaller than
when participants are passive receivers of information (Díaz et al., 2012; Lobo et al., 2018).
These results may be related to similar findings in regular (not substituted) sense modalities
such as vision (Bingham & Stassen, 1994; Mantel et al., 2015) or dynamic touch (Solomon &
Turvey, 1988; Travieso et al., 2020).
Absolute angular errors of 1.38º were observed in our experiment, on average. Such
deviations practically match the 1.4º obtained by Lobo et al. (2018). They are smaller than
the 10º to 15º obtained by Faugloire and Lejeune (2014), and the 3.7º reported by Kappers
and Koenderink (1999) for pointing with a bar to a previously touched position. The direction
of the targets did not significantly affect the absolute errors. In our interpretation, the small
size of the errors and the absence of an effect of target direction are due to the coupling
between the exploratory movements and the vibrotactile stimulation. Without this coupling,
one should expect the biases that are observed in the haptic perception of spatial relations,
such as larger errors for oblique orientations (Fernández-Díaz & Travieso, 2011; Kappers &
Koenderink, 1999; Van Mier, 2019). In our experiment, proprioceptive information was
provided together with a real-time (vibrotactile) perception-action coupling. As a
consequence, the typical biases were not observed.
The average signed error was -0.43º, which implies small leftward (counterclockwise)
deviations. Leftward errors were also observed by Lobo et al. (2018), but not by Kappers
(2018), who observed rightward errors. Our rightmost condition showed the largest
deviations and the leftmost condition the smallest. We did not expect a leftward bias, nor did
we expect differences among target directions. In the experiment, all targets were located to
the left of the hand, except the rightmost one. A vertical rod was used as a pivot that allowed
only rotations. It is possible that, with this set-up, pointing to targets on the left was slightly
more comfortable than pointing to targets on the right, which may have led to the observed
pattern of errors. In future experiments, it would be interesting to examine performance
without the biomechanical restrictions imposed by a pivot. Movements with less restrictions,
and hence less constrained perception-action couplings, may result in yet more accurate
performance.
4. Experiment 2: Moving to the Object
Experiment 1 showed that our glove allows the detection and localization of targets in
different directions. A next step is to study whether the SSD allows users to reach to and
grasp objects. We hence performed an experiment in which participants were asked to grasp
cylinders at different distances and with different sizes, but placed in the same direction
(Castiello et al., 1993; Chieffi & Gentilucci, 1993; Paulignan et al., 1991a; 1991b; Smeets &
Brenner, 1999; Zaal & Bongers, 2014). Vibration motors were attached to the back of the
index finger and thumb. The motors vibrated according to the distance to objects in the
pointing direction of the respective fingers.
The visuomotor channels model (Jeannerod, 1984) and the double-pointing model
(Smeets & Brenner, 1999) were both developed for visually-controlled prehension. One
should expect different movements for grasping based on vibrotactile information. For
example, our device requires active exploration with the hand and fingers. No vibration
occurs when the fingers do not point in the direction of the object, meaning that the distance
and size of the object are not (fully) available before the action starts. During the action,
oscillatory movements of the hand and fingers may be used to detect the direction of the
object and its edges. Such exploratory movements are not needed under visual control.
In addition to the hypothesized differences between visually-controlled and SSD-
based prehension, it is interesting to explore possible similarities. Similarities between

unimanual and bimanual reach-to-grasp movements have revealed that the control of
prehension is at least partly effector independent (Tresilian, et. al., 1997). Likewise, some
aspects of reaching and grasping may be invariant even with regard to the information that is
used to control the movements (e.g., visual or vibrotactile). One may, for example, expect the
maximum grip aperture to be scaled to the diameter of the object in both cases.
4.1 Method
A total of 42 participants performed this experiment (39 female, 3 male), with ages
between 18 and 24 years (M = 19.3, SD = 1.5). The distance and size of the target were
manipulated independently as within-subject variables (Figure 5). The object was situated at
a distance of 20, 25, or 30 cm, and its diameter was 3.0, 6.0, or 9.0 cm. The direction of the
object with regard to the starting point did not vary. Participants were not asked to grasp the
objects in a particular way (i.e., with a precision grip or a whole-hand grip).
The factorial combination of distance and size led to 3 × 3 = 9 conditions. Given that
each condition was performed five times, the experiment consisted of 9 × 5 = 45
experimental trials. In the practice phase, two distances (22.5 or 27.5 cm) and two diameters
(4.0 or 8.0 cm) were used. The target direction was the same as in the experimental phase.
With four conditions and two repetitions, participants performed 4 × 2 = 8 training trials. The
experiment was performed in a single session of approximately 30 min.
While participants performed the task with our device, in most cases we observed two
qualitatively distinct and apparently sequential phases: a reaching phase and a grasping phase
(Figure 6a). During the reaching phase, the hand points in the direction of the object and is
moved as a whole using the increase in vibration to detect the distance to the object. In
addition, the fingers tend to perform relatively small oscillations to scan the edges of the
object through the resulting on-off vibration, and thereby detect the width of the object. The
grasping phase begins when the width of the object has been detected and participants
consider that they are close enough to the object. During the grasping phase, the distance
between fingers increases to prepare the physical interaction with the object. We
computationally separated these phases to clarify several aspects of the results.
The separation of the reaching and grasping phases was performed only for trials in
which participants correctly grasped the object. The separation relied on two algorithms that
were used in combination with the visual inspection of individual time-series. The first
algorithm calculated the average speed of the index finger and thumb in the y direction and
identified the local minimum before the absolute maximum of this average y speed as the
initiation of the grasp (Figure 6b). When the visual inspection showed that this first algorithm
was not successful, a second algorithm was used. The second algorithm identified the onset
of the increase in the separation of the fingers toward the global maximum of the finger
separation as the initiation of the grasp (Figure 6c). By using the algorithms in this order, we
were able to successfully split 72.7% of the trials with successful grasps into two phases. The
first algorithm was used in 73.6% of these trials and second one in the remaining 26.4%.
We considered the following dependent variables: (a) the proportion of trials in which
the object was grasped successfully, (b) the maximum hand aperture (i.e., the maximum x
distance between the index finger and thumb), (c) the number and amplitude of the
oscillations in the hand aperture, (d) the oscillations for each finger, and (e) the covered
distance, movement duration, and speeds in the different movement phases. To identify
oscillations we used a similar procedure as in Experiment 1. The type of finger was used as a
within-subject factor in the analysis of some dependent variables.
4.2 Results
4.2.1 Preliminary analyses
Twelve trials were discarded due to recording problems (0.63% of all trials). A trial in
which the object was correctly grasped is illustrated in Figure 6a. This example shows that
the finger trajectories did not follow the smooth paths that would be predicted by models of
visually-controlled grasping. Instead, participants provoked on-off activation of the coin
motors by opening and closing the hand while pointing in the object's direction. In the shown
trial this happened most particularly in the first half of the reaching phase. Note that for the
hand aperture as a whole to oscillate, the index finger and thumb often moved in a phase-
coupled manner, meaning that when the index finger moved in one direction, the thumb
moved in the opposite direction.
4.2.2 Proportion of correct grasps
The overall proportion of successful grasps was 0.83 (SD = 0.17). A repeated-measures
ANOVA on the proportion of successful grasps, using the distance and size of the target as
factors, revealed a significant main effect of size, F(1.3, 82) = 41.56, p < .001, ηp2 = .50, and
a significant interaction, F(2.6, 164) = 3.14, p = .035, ηp2 = .07. Post hoc comparisons
showed that the smallest object was successfully grasped significantly (p < .001) fewer times
(M = 0.66, SD = 0.26) than the other two (M = 0.91, SD = 0.14). This difference was larger (p
< .05) for the shortest distance (M = 0.63, SD = 0.05) than for the longest one (M = 0.73, SD
= 0.04). For the remainder of the analysis, only trials in which the object was correctly
grasped were considered.
4.2.3 Maximum hand aperture
A repeated-measures ANOVA on the maximum hand aperture with distance and size as
factors revealed that the effect of size was the only significant effect, F(2, 358) = 138.0, p <
.001, ηp2 = .43. Bonferroni post hoc tests (p <.001) confirmed that the largest apertures were
observed for the large object (M = 10.96 cm, SD = 0.86), followed by the middle object (M =
8.91 cm, SD = 1.33), and the small one (M = 8.05 cm, SD = 1.90). The opposite pattern was
observed for the maximum hand aperture divided by the diameter of the target, F(2, 358) =
505.4, p < .001, ηp2 = .74. For the small, medium, and large objects, the hand was opened
2.68 (SD = 0.63), 1.49 (SD = 0.22), and 1.22 (SD = 0.09) times the width of the object,
respectively. We also counted in which phase the maximum apertures were observed, which
is to say, whether the last maximum in the time series of the hand opening was the largest
one. The hand reached its maximum opening during the grasping phase in 78.0% of the trials
and in the reaching phase in the remaining 22.0%. Figure 7 shows the average finger
positions per object width at the end of the trial. Obviously, the final hand aperture was larger
for larger objects.
4.2.4 Oscillations
We performed two repeated-measures ANOVAs with distance and size as factors, one
on the number of oscillations in hand aperture and one on the mean amplitude of the
oscillations. Descriptive results related to these analyses are given in Table 1. The main effect
of distance was significant for both dependent variables: F(2, 354) = 43.38, p < .001, ηp2 =
.20, for the number of oscillations, and F(2, 354) = 5.55, p < .005, ηp2 = .03, for amplitude.
The farther the object, the higher the number of oscillations (p < .001). For amplitude, only
the farthest object differed significantly from the others, showing wider oscillations (p < .05).
The main effect of size was significant only for the number of oscillations: F(2, 354) = 46.25,
p < .001, ηp2 = .21. The smaller the object, the higher the number of oscillations (p < .01).
The interaction was also significant for the number of oscillations, F(2, 354) = 4.58, p < .01,
ηp2 = .05; the difference for the medium and large objects reached significance only when the
objects were placed at the largest distance. We analyzed the differences between the
oscillations performed by the index finger and thumb with paired t tests. These tests revealed
(p < .01) that the index finger performed more oscillations than the thumb (M = 4.01 vs. M =
3.88). The amplitude of the oscillations, however, was lower (p < .001) for the index finger
than for the thumb (M = 2.64 vs. M = 3.04).
4.2.5 Reaching and grasping phases

The average overall distance that was covered in the horizontal plane was 47.28 cm (SD
= 20.12). Of that distance, 82.1% was covered during the reaching phase. The observed
velocity was higher during the grasping phase (M = 5.69 cm/s, SD = 4.23) than during the
reaching phase (M = 2.78 cm/s, SD = 1.96). On average, participants used 11.64 s (SD =
6.99) to grasp the object. The movement time was divided in 86.5% (SD = 0.07) for the
reaching phase and 13.5% (SD = .007) for the grasping phase. The oscillations were
performed almost exclusively during the reaching phase.
4.3 Discussion
The overall high proportion of successful grasps indicates that participants were able
to detect the distance and size of the objects with the glove. Objects placed at different
distances were grasped with approximately the same level of accuracy. In contrast, the
proportion of correct grasps was higher for the medium and large targets than for the small
one. For a practical purpose, the ideal result would have been to observe the same (high) level
of accuracy in all conditions. In addition to the lower proportion of successful grasps, the
small object was associated with more and larger oscillations.
Why was the small object more difficult to grasp? Given its diameter, the small target
had a narrow base of support, and hence less stability. This may be related to the fact that this
cylinder was knocked down more frequently. Perhaps for this reason, a wider relative grip
was observed for the small target than for the larger ones (2.68 vs. 1.49 and 1.22 times the
diameter of the object). Wider grips have been associated to larger safety margins (Castiello
et al., 1993; Jakobson & Goodale, 1991). With our SSD, however, opening the hand more has
the disadvantage that the fingers point less to the target, hence reducing the amount of
information that is detected. Using such a strategy may therefore lead to a more abrupt
contact between the palm of the hand and the object. In our opinion, further experiments are
needed to clarify whether the less accurate performance for the small target was a
consequence of the design of the SSD, the absence of training, or other factors. Despite the
less accurate performance for the small target, we believe that the overall level of successful
grasps in this experiment is in line with the relatively successful performance that has been
reported with a substantial number of other SSDs (Kilian et al., 2022; Lobo et al., 2014; van
Erp et al., 2005).
One of the aims of the present experiment was to tentatively explore differences and
similarities with visually-controlled prehension. In our analysis, we used two movement
phases, using labels proposed by the visuomotor channels model (Jeannerod, 1984). To some
extent consistent with this model, it seemed rather natural for participants to divide the action
into these two phases, and our algorithms successfully distinguished them. The grasp also
showed larger grip apertures for wider targets, which is one of the most characteristic finding
in the literature on the visuomotor channels model (Bongers et al., 2012; Jeannerod, 1984,
1999; Paulignan et al., 1991a, 1991b). Particularly obvious differences with visually-
controlled prehension were noted during the reaching phase. Throughout this phase,
exploratory oscillations were observed, allowing participants to detect the information needed
to control the reach and the main part of the grasp. For that reason, the hand was not
transported to the object quickly and on an approximately straight trajectory, and the grasping
phase was not initiated during the reaching phase as is common in standard prehension
patterns (Bongers et al., 2012).
In contrast to the reach-versus-grasp distinction of the visuomotor channels model
(Jeannerod, 1984), the double-pointing model (Smeets & Brenner, 1999) was less helpful in
our analysis. Two observations illustrate this. First, the considered reach and grasp
components were controlled in qualitatively different ways, making it difficult to explain the
trajectories without explicitly separating the components. Second, the oscillations of the
index finger and thumb during the reach were often performed in a phase-coupled manner.
This indicates that the hand opening was controlled as a whole, and hence that the trajectories
for the index finger and thumb were not computed independently (Smeets et al., 2019).
It is tempting, finally, to relate the different exploration patterns of the index finger
and the thumb to previous findings by Wing and Fraser (1983), who demonstrated more
stable trajectories for the thumb than for the index finger and therefore argued that the thumb
is central in the control of the reach. Although the thumb showed fewer oscillations in our
experiment, we do not believe that our findings are in line with Wing and Fraser’s (1983)
results. This is so because the thumb oscillations were larger and because the thumb still
oscillated almost as often as the index finger (3.88 vs. 4.01 oscillations per trial). Instead, our
findings may be related to the fact that the index finger has more degrees of freedom and is
therefore better suited to more numerous and more precise oscillations.
5. Experiment 3: Orienting and Moving to the Object
As demonstrated in the previous experiments, our vibrotactile glove facilitates
pointing to objects in different directions (Experiment 1) and grasping objects at different
distances and of different sizes (Experiment 2). In real life, however, objects are not all
placed at the same distance nor in the same direction. Participants in Experiment 3 were
therefore asked to grasp objects at different distances, of different sizes, and in different
directions relative to the hand. Under visual control, a simple glance allows individuals to
localize objects in space. The glove, in contrast, requires active exploration, or scanning, of
the space in which the objects are located. One may therefore hypothesize that the actual
reaching and grasping in this experiment is preceded by a phase of oscillatory exploration
that continues until the direction of the target has been detected.
The initial exploration that we expect in this experiment may be compared to the
exploration that was performed by participants in Experiment 1. In the first experiment, a
vertical rod was used as a pivot that did not permit translational movements, hence allowing
only rotational exploration. In this third experiment, movements were not restricted. This
allows us to study whether participants prefer to localize the objects with rotational
movements only (as in Experiment 1) or with an additional translational component. The
initial orientation in this experiment may therefore be quicker and easier than the exploration
in Experiment 1.
After the orientation phase, which is to say, whenever the direction of the object has
been determined, it seems reasonable to expect reach-to-grasp movements similar to the ones
observed in Experiment 2. For this reason, we expect to replicate several results of
Experiment 2, including the dependence of the maximum hand aperture on the diameter of
the object, the oscillatory exploration during the reach, and the relatively late moment of the
grasp initiation as compared to visually-controlled actions.
5.1 Method
Forty-eight participants took part in this experiment (46 female, 2 male). Their ages
ranged between 18 and 26 years (M = 19.3, SD = 1.3). As can be seen in Figure 8, three
within-subject factors determined the size and location of the object: distance to the starting
position of the hand (20 or 30 cm), diameter (5.0 or 7.0 cm), and direction with regard to the
starting position (from 90º to 150º in steps of 20º).
The used combination of factors led to 16 conditions (2 distances × 2 diameters × 4
directions). As in the previous experiments, each condition was repeated five times, leading
to 5 × 16 = 80 experimental trials. In the training phase, two distances (22 or 27 cm), two
diameters (4.0 or 8.0 cm), and three directions (100º, 120º, or 140º) were used. From these 12
conditions, eight trials were chosen randomly for the training of each participant. The
experiment was performed in one session of approximately one hour.
As in Experiment 2, the movements observed in Experiment 3 mostly consisted of
qualitatively distinct phases. The two phases that were analyzed in Experiment 2—reaching
and grasping—were also observed in this experiment (Figure 9). We separated these phases
with the same algorithms as in the previous experiment. A total of 72.5% of all trials were
successfully grasped and successfully split into phases. The first algorithm (based on y speed)
was used in 84.3% of these trials and the second algorithm (based on hand aperture) in
15.7%.
As anticipated in the introduction of this experiment, the observed movements included
an orientation phase (Figure 9a). During this initial phase, participants performed relatively
large displacements (i.e., movements in the x direction) and detected the resulting on-off
vibration to detect the direction of the object. Once the object was localized, the reaching
phase was initiated and the x speed tended to stabilize at a steadier state (which we defined as
the median speed value). To separate the orientation phase from the reaching phase, we used
an algorithm that identified the end of the final main peak in the x speed before this speed
stabilized (Figure 9c). Visual inspection of the trajectories indicated that the algorithm
correctly separated the first two phases for all of the valid trials (i.e., the trials with successful
grasps and clearly distinguished reaching and grasping phases).
Given that we wanted to compare the results of this experiment to the results observed
in the previous experiments, when possible we measured and computed similar dependent
variables. Also as in Experiment 2, we used the type of finger (index or thumb) as a within-
subject factor to examine whether there were finger-related differences in the movements.
5.2 Results
5.2.1 Preliminary analyses
Thirteen trials (0.34%) were discarded due to recording problems. An example of a trial
is shown in Figure 9.
5.2.2 Proportion of correct grasps

Overall, the proportion of successfully grasped objects was 0.93 (SD = 0.12). We
performed a repeated-measures ANOVA on the proportion of successful grasps with
distance, size, and direction as factors. The only effect that reached significance was the main
effect of size, F(1,47) = 15.31, p < .001, ηp2 = .25. The proportion of correct grasps was lower
for the smaller target (M = 0.91, SD = 0.31) than for the larger one (M = 0.95, SD = 0.24).
The following analyses were performed using only the trials with correctly grasped objects.
5.2.3 Maximum hand aperture
A t test for independent samples on the maximum distance between the tip of the index
finger and the thumb, for the smaller and larger objects, revealed a significant effect, t(47) =
5.85, p < .001. The maximum hand aperture was wider for the larger object (M = 10.00 cm,
SD = 1.23) than for the smaller one (M = 9.48 cm, SD = 1.90). As was the case in Experiment
2, when computed relative to the object size, the maximum aperture of the hand was also
different for the differently sized objects, t(47) = 20.35, p < .001. For the smaller object, the
maximum hand aperture was 1.89 (SD = 0.30) times the diameter. For the larger object, this
relative maximum was 1.43 (SD = 0.17). The maximum hand aperture occurred during the
orientation phase in 5.6% of the valid trials, during the reaching phase in 12.7% of those
trials, and during the grasping phase in the remaining 81.7%. Figure 10 shows the average
final position per finger for each experimental condition.
5.2.4 Oscillations
Table 2 summarizes the results of two repeated-measures ANOVAs using distance,
size, and direction as factors, either with the number of oscillations or with the amplitude of
the oscillations as dependent variable. Descriptive results that correspond to these analyses
are given in Table 3. Both ANOVAs revealed a significant main effect of distance. For the
longer distance, more but narrower oscillations were seen. The effect of size was significant
only for the number of oscillations. More oscillations were performed for the smaller object.
The main effect of direction was significant for both dependent variables. Participants tended
to perform fewer and narrower oscillations for more leftward targets. Concerning the
analyses per finger, we observed that the index finger performed more oscillations than the
thumb (M = 4.74 vs. M = 4.52; t[47] = 4.84, p < .001), but that these oscillations were
narrower (M = 3.47 vs. M = 3.84; t[47] = 3.55, p < .01).
5.2.5 Movement phases
The average total distance that was covered during the trials was 68.6 cm (SD = 34.8).
Of this total distance, 40.5% (SD = 9.78) was covered during the orientation phase, 41.5%
(SD = 9.19) during the reaching phase, and 18.0% (SD = 7.62) during the grasping phase. The
mean hand velocity across the successive phases was 2.58 (SD = 2.24), 2.93 (SD = 2.02), and
6.88 cm/s (SD = 3.95). The average time that was used to successfully grasp the object was
10.84 s (SD = 6.34). Of the total movement time, 35.2% (SD = 5.29) corresponded to the
orientation phase, 51.6% (SD = 8.85) to the reaching phase, and 13.2% (SD = 6.38) to the
grasping phase. At the beginning of the action, participants oriented the hand to the object
performing an average of 2.34 (SD = 0.89) oscillations, with a mean amplitude of 4.38 cm
(SD = 1.29). Then, they moved toward the object, performing 2.42 (SD = 1.25) oscillations
with a mean amplitude of 3.07 cm (SD = 0.93). Finally, the grasp was performed, usually
consisting in a single opening and closing of the hand.
5.3 Discussion
The overall proportion of correctly grasped objects in this experiment was 0.93. This
proportion is similar to the one of 0.91 for medium and large targets in Experiment 2 (with
diameters of 6.0 and 9.0 cm). In the present experiment, the targets had diameters of 5.0 and
7.0 cm. Of these, the smaller one was more difficult to grasp, possibly because of its smaller
surface of support. The distance and direction of the targets did not significantly affect the
proportion of correct grasps. This implies that the vibrotactile information about these
properties was successfully detected.
As in Experiment 2, the maximum grip aperture was wider for larger objects, and the
opposite pattern was seen when the hand opening was computed relative to the diameter of
the target. The smaller relative hand opening for larger objects may be related to physical
limitations. In fact, for the larger objects, a substantial number of participants came close to
their maximum hand aperture due to the size of their hand, leaving little room for relative
grip ratios that would match the ones seen for smaller objects. These findings are consistent
with other studies that analyzed the relation between maximum hand aperture and object size
because these studies reported positive slopes that had values of less than one (Marteniuk et
al., 1990; Smeets & Brenner, 1999; Smeets et al., 2019).
The oscillatory movements in this experiment were also consistent with the ones in
Experiment 2. For example, the index finger performed more oscillations than the thumb, but
these oscillations were narrower. With regard to the effect of direction, more oscillations
were observed for further rightward targets. A possible explanation for this effect is that, in
many cases, participants started the exploration with left-to-right movements. For this reason,
further leftward targets tended to be detected quicker, and, therefore, with fewer oscillations.
In this experiment, we observed that SSD-based prehension can be divided into three
phases. The first one, the orientation phase, was performed before the reaching and grasping
phases, which were also observed in Experiment 2 and are typical as well for visually
controlled prehension (Jeannerod, 1984). The orientation phase lasted 35.2% of the
movement time. This is less than the relative duration of the reaching phase (51.6%), but
substantially more than the duration of the grasping phase (13.2%), hence indicating the
importance of the orientation phase in the SSD-based prehension.
6. General Discussion
In this study, we addressed the reaching and grasping of objects with a sensory
substitution glove. Three experiments were performed. In the first experiment, the hand was
oriented toward objects in different directions. In the second experiment, objects at different
distances and of different sizes were actually grasped. In the third experiment, objects were
also actually grasped, but with manipulations of the initial direction of the targets in addition
to distance and size. The design of the experiments was partly inspired by a previous study by
Lobo et al. (2018), who performed a similar sequence of three experiments. Rather than
grasping objects with a 2-motor SSD on the hand, these authors addressed orienting and
locomoting toward targets with a 72-motor SSD on the abdomen. Notwithstanding the
differences in the used SSD and task, our main conclusions coincide with the ones of Lobo et
al.: SSD users spontaneously perform exploratory oscillations almost throughout the
performed actions, and this active information detection allows accurate performance.
To the best of our knowledge, the present study is the first one that tests whether a
vibrotactile glove allows the grasping of real objects with manipulations of the direction,
distance, and size of the targets. Previous SSDs that were designed to facilitate reaching
directly signaled to-be-performed actions to users (Satpute et al., 2019; Yu et al., 2016),
hence omitting the in our view central step of active information detection. Notwithstanding
these differences, given that the empirical evidence about SSD-based reaching and grasping
is (too) limited, we believe that at present it would be worthwhile to advance, and possibly
compare, all research lines that share the aim to facilitate SSD-based prehension.
We next consider the relation of our results with the visuomotor channels model
(Jeannerod, 1984). First, in Experiments 2 and 3, we observed that the maximum hand
aperture was wider for larger targets, which is one of the most consistent results of research
on grasping in general and of research on the visuomotor channels model in particular
(Jakobson & Goodale, 1991; Marteniuk et al., 1990). Second, we found it useful to
distinguish reach and grasp components in our analysis. Given that these components are key
elements of Jeannerod’s model, this may be interpreted as (admittedly indirect) support for
that model. Rather than unfolding in parallel, however, in our case the reach and grasp
components were sequential. Third, in many cases the index finger and the thumb showed
phase-related oscillatory movements during the reach, leading to a coordinated opening and
closing of the hand. One may argue that this is more consistent with the view that the hand
opening is controlled as a whole (Jeannerod, 1984) than with the view that the index finger
and the thumb are coordinated as independent units (Smeets & Brenner, 1999).
The late initiation of the grasp in our experiments is at least partly related to the fact
that, during the reach, the fingers kept pointing (intermittently) toward the target in order to
maintain the on-off pattern of vibration. In addition, interesting parallels can be noted with
previous research. Haggard and Wing (1998) demonstrated that the hand opening is delayed
for curved movements. Citing research by Broadbent (1982), they argued that this is related
to the difficulty of curved movements as compared to straight ones. In their words, the
“delayed hand opening in the curved movements … may reflect a shift towards sequential
performance due to increased complexity” (p. 291). In addition to the study by Haggard and
Wing, Bongers et al. (2012) identified several other studies in which the initiation of the
grasp was delayed with respect to the standard hand opening pattern. Bongers et al.
performed an experiment to further investigate such apparently non-standard grasp patterns.
In their experiment, objects were placed at farther distances than is common in the literature
on reaching and grasping. Several non-standard prehension patterns were observed, including
ones with delays in the hand opening. Our results, with sequential reaching and grasping,
may be interpreted as examples of such non-standard patterns.
Another difference with standard prehension patterns that we observed is the initial
movement phase in our Experiment 3. In that experiment, individuals used about one third of
the movement time to detect the direction of the targets, in a phase that we referred to as
orientation phase. Based on the data presented in this article and on our own experience as
users of the SSD, we believe that the transition from the orientation phase to the reaching
phase is often quite abrupt. The orientation phase ends and the reaching phase begins at the
moment that the user becomes aware of the target direction. During the reaching phase, the
existing awareness of the target direction is continuously updated due to the on-off
stimulation. In sum, the observed movements consisted of three sequential phases: orienting,
reaching, and grasping. To emphasize the importance of the first phase, we suggest to refer to
the movements with the term orient-to-reach-to-grasp movements. This in contrast to the
term reach-to-grasp movements that is common in the literature on visually-controlled
grasping (Smeets et al., 2019; Tresilian, et. al., 1997).
Our findings should also be discussed in relation to the debate about real-time
information usage in prehension (Castiello et al., 1993; Jakobson & Goodale, 1991; Wing et
al., 1986). It has been argued that prehension, and in particular the reaching phase of
prehension, can to a large extent be performed in a pre-programmed manner (Jeannerod,
1999; without denying possible modifications in case of unexpected perturbations, Paulignan
et al., 1991a, 1991b). However, real-time information-based control seems better suited to
explain the SSD-based movements in our study (Kolarik et al., 2017; Lobo et al., 2019). This
is so because the continuous exploration makes sense only in the context of a close relation
between perception and action.
The here-presented glove and our research with it have several limitations. For
example, we do not know to what extent objects with different shapes and affordances can be
detected, identified, and grasped (Newell et al., 1989). Also, the detection range of the glove
is short and only a single target was considered at each time. In this sense, it may be
interesting to speculate about the combined use of this SSD for the prehension of nearby
objects and other special purpose SSDs for the interaction with objects or obstacles slightly
farther away (Lobo et al., 2018). Another limitation of our research is the fact that individuals
with visual impairments (i.e., potential users of the device) have not been involved (Barontini
et al., 2020). It might be advisable, for further studies and developments, to perform
preliminary research concerning their opinions, considerations, and needs, and the main
difficulties that they encounter in their daily life. In a similar vein, it would be desirable to
move from laboratory studies to field studies in more natural environments (Merino et al.,
2020). Unfortunately, field studies are not yet possible with the current version of our SSD,
because the device depends on the external movement registration system. Having said this,
from a technological point of view it seems feasible to develop a glove that is similar to ours
but that does not depend on the movement registration. A recently presented SSD glove is
substantially further advanced in this regard (Kilian et al., 2022). That glove, however, is
sensitive to objects at farther distances and is meant for navigation rather than for reaching
and grasping.
To conclude, several authors in the field of sensory substitution have demonstrated that
special-purpose devices allow users to perceive and act upon affordances of the environment
(Warren, 1984; Warren & Whang, 1987). Affordances that have been considered include the
possibility of safe navigation while avoiding obstacles (Kilian et al., 2022; Lobo et al., 2019),
the passability of apertures (Favela et al., 2018; Kolarik et al., 2014; de Paz et al., 2019), and
the climbability of obstacles (Lobo et al., 2014; Travieso et al., 2105). In this particular study,
we addressed the graspability of objects in peripersonal space. We believe that, as researchers
include more daily-life tasks in the study of SSDs, and improve the technology of the devices
to make them more reliable and robust, we may get closer to the final goal of research on
SSDs: a more general use of the devices by potential users.

7. Funding Sources
This research was supported in part by funds from the H2020 Marie Sklodowska-Curie
Innovation Training Network programme, funded by the European Union: Project REPAIRS.
Grant number 956003.
Credit author statement
Carlos de Paz: Conceptualization, Methodology, Software, Investigation, Writing- Original draft

preparation
Jorge Ibáñez-Gijón: Conceptualization, Data curation, Software, Visualization
David Travieso: Conceptualization, Methodology, Validation, Supervision, Resources, Writing-

Reviewing and Editing
David M. Jacobs: Conceptualization, Visualization, Validation, Supervision, Resources, Writing-

Reviewing and Editing
Declaration of interests
☒The authors declare that they have no known competing financial interests or personal
relationships that could have appeared to influence the work reported in this paper.
8. References
Bach-y-Rita, P., Collins, C.C., Saunders, F.A., White, B., Scadden, L., 1969. Vision
substitution by tactile image projection. Nature. 221, 963.
https://doi.org/10.1038/221963a0
Barontini, F., Catalano, M.G., Pallottino, L., Leporini, B., Bianchi, M., 2020. Integrating
wearable haptics and obstacle avoidance for the visually impaired in indoor
navigation: A user-centered approach. IEEE Trans. Haptics. 14, 109-122.
https://doi.org/10.1109/TOH.2020.2996748
Bingham, G.P., Stassen, M.G., 1994. Monocular egocentric distance information generated
by head movement. Ecol. Psychol. 6, 219–238.
http://dx.doi.org/10.1207/s15326969eco0603_4
Bongers, R.M., Zaal, F.T., Jeannerod, M., 2012. Hand aperture patterns in prehension. Hum.
Mov. Sci. 31(3), 487-501. https://doi.org/10.1016/j.humov.2011.07.014
Bourne, R.R., Flaxman, S.R., Braithwaite, T., Cicinelli, M.V., Das, A., Jonas, J. B., Naidoo,
K., 2017. Magnitude, temporal trends, and projections of the global prevalence of
blindness and distance and near vision impairment: a systematic review and meta-
analysis. Lancet Glob. Health. 5, e888-e897. https://doi.org/10.1016/S2214-
109X(17)30293-0
Bozzacchi, C., Brenner, E., Smeets, J.B., Volcic, R., Domini, F., 2018. How removing visual
information affects grasping movements. Exp. Brain Res. 236, 985-995.
https://doi.org/10.1007/s00221-018-5186-6
Broadbent, D.E., 1982. Task combination and selective intake of information. Acta Psychol.
50, 253-290. https://doi.org/10.1016/0001-6918(82)90043-9

Cancar, L., Díaz, A., Barrientos, A., Travieso, D., Jacobs, D.M., 2013. Tactile-sight: A
sensory substitution device based on distance-related vibrotactile flow. Int. J. Adv.
Robot. Syst. 10, 272. https://doi.org/10.5772/56235
Castiello, U., Bennett, K. M.B., Mucignat, C., 1993. The reach to grasp movement of blind
subjects. Exp. Brain Res. 96, 152-162. https://doi.org/10.1007/BF00230448
Chieffi, S., Gentilucci, M., 1993. Coordination between the transport and the grasp
components during prehension movements. Exp. Brain Res. 94, 471-477.
https://doi.org/10.1007/BF00230205
De Paz, C., Travieso, D., Ibáñez-Gijón, J., Bravo, M., Lobo, L., Jacobs, D.M., 2019. Sensory
substitution: The affordance of passability, body-scaled perception, and exploratory
movements. PLoS One. 14, e0213342. https://doi.org/10.1371/journal.pone.0213342
Díaz, A., Barrientos, A., Jacobs, D.M., Travieso, D., 2012. Action-contingent vibrotactile
flow facilitates the detection of ground level obstacles with a partly virtual sensory
substitution device. Hum. Mov. Sci. 31, 1571-1584.
https://doi.org/10.1016/j.humov.2012.05.006
Faugloire, E., Lejeune, L., 2014. Evaluation of heading performance with vibrotactile
guidance: The benefits of information–movement coupling compared with spatial
language. J. Exp. Psychol.-Appl. 20, 397-410. https://doi.org/10.1037/xap0000032
Faul, F., Erdfelder, E., Lang, A.G., Buchner, A., 2007. G*Power 3: A flexible statistical
power analysis program for the social, behavioral, and biomedical sciences. Behav.
Res. Methods. 39, 175-191. https://doi.org/10.3758/BF03193146
Favela, L.H., Riley, M.A., Shockley, K., Chemero, A., 2018. Perceptually equivalent
judgments made visually and via haptic sensory-substitution devices. Ecol. Psychol.
30, 326-345. https://doi.org/10.1080/10407413.2018.1473712

Fernández-Díaz, M., Travieso, D., 2011. Performance in haptic geometrical matching tasks
depends on movement and position of the arms. Acta Psychol. 136, 382-389.
https://doi.org/10.1016/j.actpsy.2011.01.003.
Froese, T., McGann, M., Bigge, W., Spiers, A., Seth, A.K., 2012. The enactive torch: a new
tool for the science of perception. IEEE Trans. Haptics. 5, 365-375.
https://doi.org/10.1109/TOH.2011.57
Froese, T., Ortiz-Garin, G.U., 2020. Where is the action in perception? An exploratory study
with a haptic sensory substitution device. Front. Psychol. 11, 809.
https://doi.org/10.3389/fpsyg.2020.00809
Gibson, J.J., 1979. The ecological approach to visual perception. Boston, MA: Houghton
Mifflin.
Goodale, M.A., Pelisson, D., Prablanc, C., 1986. Large adjustments in visually guided
reaching do not depend on vision of the hand or perception of target
displacement. Nature. 320, 748-750. https://doi.org/10.1038/320748a0
Haggard, P., Wing, A., 1998. Coordination of hand aperture with the spatial path of hand
transport. Exp. Brain Res. 118(2), 286-292. https://doi.org/10.1007/s002210050283
Kilian, J., Neugebauer, A., Scherffig, L., Wahl, S., 2022. The Unfolding Space Glove: A
Wearable Spatio-Visual to Haptic Sensory Substitution Device for Blind
People. Sensors. 22(5), 1859. https://doi.org/10.3390/s22051859
Jakobson, L. S., Goodale, M.A., 1991. Factors affecting higher-order movement planning: a
kinematic analysis of human prehension. Exp. Brain Res. 86, 199-208.
https://doi.org/10.1007/BF00231054
Jeannerod, M., 1984. The timing of natural prehension movements. J. Mot. Behav. 16, 235-
254. https://doi.org/10.1080/00222895.1984.10735319
Jeannerod, M., 1986. The formation of finger grip during prehension. A cortically mediated
visuomotor pattern. Behav. Brain Res. 19, 99-116. https://doi.org/10.1007/978-94-
009-4462-6_10
Jeannerod, M., 1999. Visuomotor channels: Their integration in goal-directed
prehension. Hum. Mov. Sci. 18, 201-218. https://doi.org/10.1016/S0167-
9457(99)00008-1
Jeannerod, M., Arbib, M.A., Rizzolatti, G., Sakata, H., 1995. Grasping objects: The cortical
mechanisms of visuomotor transformation. Trends Neurosci. 18, 314-320.
https://doi.org/10.1016/0166-2236(95)93921-J
Jicol, C., Lloyd-Esenkaya, T., Proulx, M.J., Lange-Smith, S., Scheller, M., O'Neill, E.,
Petrini, K., 2020. Efficiency of sensory substitution devices alone and in
combination with self-motion for spatial navigation in sighted and visually impaired.
Front. Psychol. 11, 1443. https://doi.org/10.3389/fpsyg.2020.01443
Kaas, A.L., Van Mier, H.I., 2006. Haptic spatial matching in near peripersonal space. Exp.
Brain Res. 170, 403-413. https://doi.org/10.1007/s00221-005-0223-7
Kaczmarek, K.A., Webster, J.G., Bach-y-Rita, P., Tompkins, W.J., 1991. Electrotactile and
vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Eng. 38,
1-16. https://doi.org/10.1109/10.68204
Kappers, A.M., 2018. Minute hands of clocks indicating the same time are not perceived as
haptically parallel. Sci Rep. 8, 1-9. https://doi.org/10.1038/s41598-018-21415-9
Kappers, A.M., Koenderink, J.J., 1999. Haptic perception of spatial relations. Perception. 28,
781-795. https://doi.org/10.1068/p2930
Kolarik, A.J., Scarfe, A.C., Moore, B.C., Pardhan, S., 2017. Blindness enhances auditory
obstacle circumvention: Assessing echolocation, sensory substitution, and visual-

based navigation. PLoS One. 12(4), e0175750.
https://doi.org/10.1371/journal.pone.0175750
Kolarik, A.J., Timmis, M.A., Cirstea, S., Pardhan, S., 2014. Sensory substitution information
informs locomotor adjustments when walking through apertures. Exp. Brain Res.
232, 975-984. https://doi.org/10.1007/s00221-013-3809-5
Lenay, C., Canu, S., Villon, P., 1997. Technology and perception: The contribution of
sensory substitution systems. In: Proceedings Second International Conference on
Cognitive Technology Humanizing the Information Age. IEEE. 44-53.
https://doi.org/10.1109/CT.1997.617681
Lenay, C., Gapenne, O., Hanneton, S., Marque, C., Genouëlle, C., 2003. Sensory substitution:
Limits and perspectives. In: Hatwell, Y., Streri, A., Gentaz, E., (Eds.), Touching for
knowing. John Benjamins, Amsterdam, 275-292.
https://doi.org/10.1075/aicr.53.22len
Lenay, C., Steiner, P., 2010. Beyond the internalism/externalism debate: The constitution of
the space of perception. Conscious. Cogn. 19, 938-952.
https://doi.org/10.1016/j.concog.2010.06.011
Lobo, L., Nordbeck, P.C., Raja, V., Chemero, A., Riley, M.A., Jacobs, D.M., Travieso, D.,
2019. Route selection and obstacle avoidance with a short-range haptic sensory
substitution device. Int. J. Hum.-Comput. Stud. 132, 25-33.
https://doi.org/10.1016/j.ijhcs.2019.03.004
Lobo, L., Travieso, D., Barrientos, A., Jacobs, D.M., 2014. Stepping on obstacles with a
sensory substitution device on the lower leg: Practice without vision is more
beneficial than practice with vision. PLoS One. 9, e98801.
Lobo, L., Travieso, D., Jacobs, D.M., Rodger, M., Craig, C.M., 2018. Sensory substitution:
Using a vibrotactile device to orient and walk to targets. J. Exp. Psychol.-Appl. 24,
108. https://doi.org/10.1037/xap0000154
Loomis, J.M., Klatzky, R.L. Giudice, N.A., 2012. Sensory substitution of vision: importance
of perceptual and cognitive processing. In: Manduchi, R., Kurniawan, S.
(Eds.), Assistive Technology for Blindness and Low Vision. CRC Press, Boca
Raton, FL, USA, 162-191. https://doi.org/10.1201/9781315216935-12
Mantel, B., Stoffregen, T.A., Campbell, A., Bardy, B.G., 2015. Exploratory movement
generates higher-order information that is sufficient for accurate perception of scaled
egocentric distance. PLoS One. 10(4), e0120025.
Marteniuk, R.G., Leavitt, J.L., MacKenzie, C.L., Athenes, S., 1990. Functional relationships
between grasp and transport components in a prehension task. Hum. Mov. Sci. 9(2),
149-176. https://doi.org/10.1016/0167-9457(90)90025-9
Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., Weiskopf, D., 2020.
Evaluating mixed and augmented reality: A systematic literature review (2009-
2019). In 2020 IEEE International Symposium on Mixed and Augmented Reality
(ISMAR) (pp. 438-451). IEEE. https://doi.org/10.1109/ISMAR50242.2020.00069
Newell, K.M., Cesari, P., 1999. On taking the grasping out of prehension. Motor Control. 3,
285-288. https://doi.org/10.1123/mcj.3.3.285
Newell, K.M., Scully, D.M., Tenenbaum, F., Hardiman, S., 1989. Body scale and the
development of prehension. Dev. Psychobiol. 22, 1-13.
https://doi.org/10.1002/dev.420220102
O'Regan, J.K., Noë,A., 2001. A sensorimotor account of vision and visual consciousness.
Behav. Brain Sci. 24, 939-973. https://doi.org/10.1017/S0140525X01000115

O’Shea, H., Redmond, S.J., 2021. A review of the neurobiomechanical processes underlying
secure gripping in object manipulation. Neurosci. Biobehav. Rev. 123, 286-300.
https://doi.org/10.1016/j.neubiorev.2021.01.007
Paulignan, Y., MacKenzie, C., Marteniuk, R., Jeannerod, M., 1991a. Selective perturbation of
visual input during prehension movements. I. The effects of changing the object
position. Exp. Brain Res. 83, 502-512. https://doi.org/10.1007/BF00231858
Paulignan, Y., Jeannerod, M., MacKenzie, C., Marteniuk, R., 1991b. Selective perturbation
of visual input during prehension movements. II. The effects of changing the object
size. Exp. Brain Res. 87, 407–420. https://doi.org/10.1007/BF00231858
Rizzolatti, G., Fadiga, L., Fogassi, L., Gallese, V., 1997. The space around us. Science. 277,
190-191. https://doi.org/10.1126/science.277.5323.190.
Satpute, S., Canady, J., Klatzky, R.L., Stetten, G., 2019. FingerSight: A vibrotactile wearable
ring for assistance with locating and reaching objects in peripersonal space. IEEE
Trans. Haptics. 13, 325-333. https://doi.org/10.1109/TOH.2019.2945561
Smeets, J.B., Brenner, E., 1999. A new view on grasping. Motor Control. 3, 237-271.
https://doi.org/10.1123/mcj.3.3.237
Smeets, J.B., van der Kooij, K., Brenner, E., 2019. A review of grasping as the movements of
digits in space. J. Neurophysiol. 122, 1578-1597.
https://doi.org/10.1152/jn.00123.2019.
Solomon, H.Y., Turvey, M.T., 1988. Haptically perceiving the distances reachable with hand-
held objects. J. Exp. Psychol.-Hum. Percept. Perform. 14, 404–427.
http://dx.doi.org/10.1037/0096-1523.14.3.404
Spence, C., 2014. The skin as a medium for sensory substitution. Multisens. Res. 27, 293-
312. https://doi.org/10.1163/22134808-00002452.
Starkiewicz, W., Kuliszewski, T., 1963. Active energy radiating systems: The 80-channel
elektroftalm. In: Proceedings of the international Congress on Technology and
Blindness. American Foundation for the Blind. 1, 157-166.
Stival, F., Michieletto, S., Cognolato, M., Pagello, E., Müller, H., Atzori, M., 2019. A
quantitative taxonomy of human hand grasps. J. NeuroEng. Rehabil. 16, 1-17.
https://doi.org/10.1186/s12984-019-0488-x
Travieso, D., Gómez-Jordana, L., Díaz, A., Lobo, L., Jacobs, D.M., 2015. Body-scaled
affordances in sensory substitution. Conscious. Cogn. 38, 130-138.
https://doi.org/10.1016/j.concog.2015.10.009
Travieso, D., Lobo, L., De Paz, C., Langelaar, T.E., Ibáñez-Gijón, J., Jacobs, D.M., 2020.
Dynamic touch as common ground for enactivism and ecological psychology. Front.
Psychol. 11, 1257. https://doi.org/10.3389/fpsyg.2020.01257
Tresilian, J.R., Stelmach, G.E., Adler, C.H., 1997. Stability of reach-to-grasp movement
patterns in Parkinson's disease. Brain. 120, 2093-2111.
https://doi.org/10.1093/brain/120.11.2093
van de Kamp, C., Zaal, F.T., 2007. Prehension is really reaching and grasping. Exp. Brain
Res. 182, 27-34. https://doi.org/10.1007/s00221-007-0968-2
van der Kamp, J., Savelsbergh, G.J., Davis, W.E., 1998. Body-scaled ratio as a control
parameter for prehension in 5- to 9-year-old children. Dev. Psychobiol. 33, 351-361.
https://doi.org/10.1002/(SICI)1098-2302(199812)33:4<351::AID-DEV6>3.0.CO;2-
van Erp, J.B.F., van Veen, H.A.H.C, Jansen, C., Dobbins, T., 2005. Waypoint navigation
with a vibrotactile waist belt. ACM Trans. Appl. Percept. 2, 106-117.
https://doi.org/10.1145/1060581.1060585
Van Mier, H.I., 2019. Changing the influence of the egocentric reference frame impacts
deviations in haptic parallelity matching. Exp. Brain Res. 237, 2387-2395.
https://doi.org/10.1007/s00221-019-05596-x.
Verheij, R., Brenner, E., Smeets, J.B., 2012. Grasping kinematics from the perspective of the
individual digits: a modelling study. PLoS One. 7, e33150.
Wallace, S.A., Weeks, D.L., 1988. Temporal constraints in the control of prehensile
movement. J. Mot. Behav. 20, 81-105.
https://doi.org/10.1080/00222895.1988.10735435
Warren, W.H., 1984. Perceiving affordances: Visual guidance of stair climbing. J. Exp.
Psychol.-Hum. Percept. Perform. 10, 683-703. https://doi.org/10.1037/0096-
1523.10.5.683
Warren, W.H., Whang, S., 1987. Visual guidance of walking through apertures: body-scaled
information for affordances. J. Exp. Psychol.-Hum. Percept. Perform. 13, 371-383.
https://doi.org/10.1037/0096-1523.13.3.371
Wing, A.M., Fraser, C., 1983. The contribution of the thumb to reaching movements. Q. J.
Exp. Psychol. Sect A-Hum. Exp. Psychol. 35, 297-309.
https://doi.org/10.1080/14640748308402135
Wing, A.M., Turton, A., Fraser, C., 1986. Grasp size and accuracy of approach in reaching. J.
Mot. Behav. 18, 245-260. https://doi.org/10.1080/00222895.1986.10735380
Yu, Z., Horvath, S., Delazio, A., Wang, J., Almasi, R., Klatzky, R.J., Galeotti, J., Stetten, G.,
2016. Palmsight: An assistive technology helping the blind to locate and grasp
objects. Carnegie Mellon University, Pittsburgh, PA, Tech. Rep., CMU-RI-TR-16-
59.
Zaal, F.T., Bongers, R.M., 2014. Movements of individual digits in bimanual prehension are
coupled into a grasping component. PLoS One. 9, e97790.
Tables and Figures
Table 1
Number and Amplitude of Oscillations as a Function of Distance and Size in Experiment 2.
Distance (cm) Size (cm)

20 25 30 3.0 6.0 9.0
Number of 5.76 7.44 10.45 10.75 7.26 5.86
Oscillations (#) (2.82) (4.03) (6.21) (6.24) (3.71) (3.24)
Amplitude of 2.64 2.84 3.06 2.94 2.82 2.77

Oscillations (cm) (1.04) (0.97) (1.02) (1.09) (0.97) (1.01)
Note. Values in parenthesis are standard deviations.
Table 2
Results of ANOVAs on Number and Amplitude of Oscillations in Experiment 3.
Number of Oscillations
Dis ×
Distance Size Direction Dis × Size Dis × Dir Size × Dir
Size × Dir
F 123.83 23.07 18.17 1.60 1.92 .70 2.22
p *** *** *** .21 .13 .55 .09
ηp2 .77 .38 .33 .04 .05 .02 .06
Amplitude of Oscillations
Dis ×
Distance Size Direction Dis × Size Dis × Dir Size × Dir
Size × Dir
F 12.81 0.01 5.56 13.87 2.35 1.77 0.40
p ** .91 * ** .10 .16 .63
ηp 2
.30 .00 .16 .32 .07 .05 .01
Note. * < 0.05, ** < 0.01, *** < 0.001
Table 3
Number and Amplitude of Oscillations as a Function of Distance, Size, and Direction in
Experiment 3.
Distance (cm) Size (cm) Direction (deg)

20 30 5.0 7.0 90 110 130 150
Number of 7.85 12.25 10.68 9.42 11.10 10.64 9.88 8.58

Oscillations (#) (4.00) (5.39) (5.32) (4.56) (6.54) (5.04) (4.65) (4.33)
Amplitude of 3.68 3.33 3.51 3.50 3.93 3.41 3.31 3.39

Oscillations (cm) (1.42) (1.27) (1.80) (1.92) (1.90) (1.90) (1.86) (1.77)
Note. Values in parenthesis are standard deviations.

Figure 1.
Sensory substitution glove.
Note. A) Sensory Substitution Glove. Six reflective markers are visible. The two vibration
motors are covered. B) Schematic representation of the functioning of the device. The motor
on each finger (represented in red) vibrates if the line that crosses the two markers on the
finger (in white) intersects the object.

Figure 2.
Set-up of Experiment 1.
110° 70°
130° 90°
150°
20°
Note. The five open circles represent the five targets. The white dots on the hand represent
the markers and the red square the vibration motor. A vertical rod was grabbed allowing
rotations but not displacements of the hand. Four movement-registration cameras were used.
Figure 3.
Example of a trial in Experiment 1.
Note. Shown is the x coordinate of the farthest marker on the index finger for a target located
at 90º.
Figure 4.
Signed errors in Experiment 1.
Note. The red crosses indicate average final pointing direction per target. Inserted panels are
enlargements for leftmost (150º) and rightmost (70º) conditions. The straight lines indicate
the center of the objects.

Figure 5.
60˚
Note. Three target widths were combined factorially with three distances. Four movement-
registration cameras were used. The white dots on the hand represent the markers and the red
squares the vibration motors.

Figure 6.
Note. A) Displacement of the index finger, the thumb, and their average. B) Velocity profile
for y coordinate averaged for index finger and thumb. The dot in this panel indicates the local
minimum before the global maximum (used in first algorithm; see text for details). C)
Distance between tip of index finger and thumb. The dot here indicates the moment at which
the fingers distance initiates the increase toward the global maximum (used in second
algorithm; see text for detail). In this trial, the two algorithms identified the same cut-off
point.
Figure 7.
Final finger positions in Experiment 2.
Note. The large black circumferences represent objects with diameters of 3.0, 6.0, and 9.0
cm. The black circles and squares indicate the average final positions of the tip of the index
finger and thumb, respectively. Each pair of finger-points is connected with a line segment,
labelled with the corresponding object diameter.

Figure 8.
20º
Note. Four target directions, two distances, and two diameters were combined factorially. Six
movement-registration cameras were used. The white dots on the hand represent the markers
and the red squares the vibration motors.

Figure 9.
Note. A) Displacement of the index finger, the thumb, and their average. B) Velocity profile
for y coordinate averaged for index finger and thumb. C) Velocity profile for x coordinate of
averaged finger trajectory. The reaching and grasping phases were separated as in the
previous experiment. In addition, the orientation phase was separated from the reaching phase
with an algorithm based on the end of the final main peak in the profile of the x coordinate
before stabilizing (black dot in Panel C; see text for detail).

Figure 10.
Final finger positions in Experiment 3.
Note. The final position of the fingertip is indicated with red circles for the index finger and
blue circles for the thumb. The large black circumferences represent large (left panel) and
small (right panel) objects.

Author Biographies
Carlos de Paz, is Ph.D. student at the Department of Psychology of the Universidad
Autónoma de Madrid and member of the Perception and Action Research Group.
Inspired by ecological psychology, his research investigates affordances-based
perception with sensory substitution devices and sensorimotor coupling.
Jorge Ibáñez-Gijón, Ph.D., is assistant Professor and member of the Perception and
Action Research Group at the Universidad Autónoma de Madrid. His approach
aims to account for psychological and biological phenomena as the unfolding of
complex dynamical systems in interaction with their environment. His research
interests are perception-action, sport sciences, ecological robotics, cybernetics, biosemiotics, and
theoretical biology.
David Travieso, Ph.D., is Associate Professor and Co-Director of the Perception and
Action Research Group at the Universidad Autónoma de Madrid. His research
interests are haptics and sensory substitution, as well as the theory and history of
psychology. His approach emphasizes the perception of affordances and
perception-action coupling, classic concepts in the ecological approach.
David M. Jacobs, Ph.D., is Associate Professor and Co-Director of the Perception
and Action Research Group at the Universidad Autónoma de Madrid. His main
theoretical contributions are formulated around an ecological theory of learning
that he, with Claire F. Michaels, has named direct learning. His most recent
empirical research aims to apply and advance ecological ideas about perception, action, and
learning, in the fields of sensory substitution, aviation, and sports.

Grasping Objects With A Sensory Substitution Glove

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Grasping Objects With A Sensory Substitution Glove

Uploaded by

Copyright:

Available Formats

Journal Pre-proof

Grasping Objects with a Sensory Substitution Glove

Carlos de Paz , Jorge Ibáñez-Gijón , David Travieso ,

To appear in: International Journal of Human-Computer Studies

Received date: 15 February 2022

© 2022 Published by Elsevier Ltd.

Carlos de Paz, Jorge Ibáñez-Gijón, David Travieso, David M. Jacobs

Facultad de Psicología, Universidad Autónoma de Madrid, 28049, Madrid, Spain

Running Head: Sensory Substitution and Grasping

Correspondence concerning this article should be addressed to David Travieso, Facultad de

Psicología, Universidad Autónoma de Madrid, 28049 Madrid, Spain.

information contingent on active exploration.

 Objects were successfully grasped in 88% of the trials.

blindfolded participants. Experiment 1 tested the ability of participants to point to cylindrical

(3.82º). In Experiment 2, participants aimed to grasp differently sized cylinders at different

reaching and grasping components. Oscillatory explorations were observed in all

experiments, permitting the detection of the information that is needed to successfully

complete the action.

Keywords: grasping, reach-to-grasp movement, sensory substitution, visual impairment,

visuomotor channels model

individuals with a visual impairment.

or segmental movements, or try to keep objects at the same place.

Notwithstanding the variety of compensatory strategies, developing aids for the

actions under more natural conditions.

evolved from trying to substitute vision as a whole (general-purpose devices) to trying to

Exploration with such devices permits users to establish perception-action couplings, or

1991b; Wing & Fraser, 1983).

trajectories of the thumb and index finger are controlled independently.

that underlie the double-pointing model.

In addition to the study of visually-controlled grasping, assistive devices have been

designed that provide vibrotactile information of locations in peripersonal space. Among

objects in peripersonal space.

relative to the size of the hand.

To summarize, a vibrotactile glove that is meant to facilitate grasping should ideally

The local ethics committee approved the experimental protocol (UAM-CEI-94-1717)

All participants were students at the Faculty of Psychology of the Universidad

(Faul et al., 2007).

the target locations were available.

As to-be-grasped objects, we used plastic 10-cm-high cylinders with different

a very minor increase in variability.

2.4 Functioning of SSD

In Experiment 1, the vibration intensity was fixed at 8 V. In Experiments 2 and 3, the

𝑉 = 0.009 × 𝑑2 − 0.5 × 𝑑 + 10 (1)

corresponded, approximately, to frequencies of vibration of 42 and 65 Hz (Appendix A of

and its maximum admitted voltage.

to the object, the vibration increases”.

1) or when the target was displaced (Experiments 2 and 3).

2.6 Data Acquisition, Preprocessing, and Statistical Analysis

direction than in the x direction.

We performed repeated-measures analyses of variance (ANOVAs) to test the effects of

the independent variables, after averaging data over repetitions. Greenhouser-Geisser

2.7 Transparency and openness

corresponding author. This study was not preregistered.

3. Experiment 1: Orienting the Hand to the Object

exploration (de Paz et al., 2019; Lenay et al., 2003).

plane (i.e., for oblique orientations).

deviations regardless of the target direction.

located at a distance of 30 cm (from the hand position).

3.2.1 Preliminary analysis

smaller movements to localize the center of the object.

3.2.2 Absolute angular errors

correctly detect and point to the objects.

3.2.3 Signed angular errors