404
2, APRIL-JUNE 2020
Haptic Material Analysis and Classification Inspired
by Human Exploratory Procedures
Matti Strese , Member, IEEE, Lara Brudermueller , Jonas Kirsch , and Eckehard Steinbach , Fellow, IEEE
Abstract—We present a framework for the acquisition and
parametrization of object material properties. The introduced
acquisition device, denoted as Texplorer2, is able to extract
surface material properties while a human operator is performing
exploratory procedures. Using the Texplorer2, we scanned
184 material classes which we labeled according to biological,
chemical, and geological naming conventions. Based on these real
material recordings, we introduce a novel set of mathematical
features which align with corresponding material properties
defined in perceptual studies from related work and classify the
materials using common machine learning techniques. Validation
results of the proposed multi-modal features lead to an overall
classification accuracy of 90.2%
1.2% and an F1 score of 0.90

0.01 using the random forest classifier. For the sake of
comparison, a deep neural network is trained and tested on
images of the material surfaces; it outperforms (90.7%
1.0%)
the hand-crafted feature-based approach yet leads to more critical
misclassifications in terms of the proposed taxonomy.
Index Terms—Surface Haptics, material scanning, content-
based features.
I. INTRODUCTION
T HE human haptic perception system relies on both tactile
and kinesthetic sensory information when interacting with
objects. Lederman et al. [1] summarize that humans typically
perform six exploratory procedures (EPs) to identify unknown
objects: 1) enclosure reveals global shape information about the
object which is also linked to its visual inspection. The relation
between objects’ visual and haptic representations is an impor-
tant issue in multi-sensory processing according to the insights
of Lacey et al. [2] and has also been confirmed by Vardar
et al. [3] for fingertip interaction data and visual textures.
Humans then 2) lift objects to identify their weights. In parallel,
3) static touch is used to identify thermal properties; 4) pressing
on the materials further enables assessment of the hardness.
5) Contour following and 6) lateral sliding motions allow users
to identify the contour, and also to let them feel the fine
roughness and the object surface’s friction properties. During
this procedure, humans can distinguish between microscopic
Manuscript received February 5, 2019; revised October 1, 2019; accepted
October 29, 2019. Date of publication November 8, 2019; date of current ver-
sion June 8, 2020. This paper was recommended for publication by Associate
Editor A. M. L. Kappers and Editor L. Jones upon evaluation of the reviewers’
comments. (Corresponding author: Matti Strese.)
The authors are with the Chair of Media Technology, Technical University
of Munich9184, Munich 80333, Germany (e-mail: matti.strese@tum.de; lara.
brudermueller@tum.de; jonas.kirsch@tum.de; eckehard.steinbach@tum.de).
This paper has supplementary downloadable material available at http://
ieeexplore.ieee.org, provided by the authors.
Digital Object Identifier 10.1109/TOH.2019.2952118
1939-1412 ß 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO.
and macroscopic roughness of a surface. These two tactile
dimensions received immense attention during almost a
century, beginning with the duplex theory of roughness by
Katz [4], the research studies and survey of Lederman et al. [5],
Klatzky et al. [6], Bensma€ıa et al. [7], [8], and Bergmann Tiest
et al. [9], [10]. In parallel, auditory perception during finger-
surface interaction is linked to the perception of roughness and
is cognitively fused into an overall understanding of the touched
material according to Schroeder et al. [11] and Yau et al. [12].
Altogether, these EPs lead to a set of haptic material proper-
ties which we intend to capture by manually recording object
surface interaction data using a novel human-operated object
scanner, denoted as Texplorer2. Data traces resulting from
these scans provide the input for supervised material classifica-
tion based on multi-modal features, e.g., content-based tactile
features. Additionally, such features also enable the potential
reproduction, i.e. haptic display, of the material properties.
A. Tactile Dimensions
There is still an active discussion about the feature space
describing all relevant tactile experiences [13]. By summarizing
previous studies, Okamoto et al. [14] identified the five major
tactile dimensions: friction, hardness, warmth (thermal proper-
ties), macroscopic, and microscopic roughness. Further sub-
dimensions have already been inferred by Fishel et al. (Syn-
Touch, USA) [15], who propose a 15-dimensional feature
space [15], [16]. Their definitions allow for a further subtle dis-
tinction between these five major dimensions by separating, for
example, hardness into compliance, yielding, relaxation, damp-
ing, and local deformation around the sensing device. To avoid
potential confusion with the features defined by Fishel
et al. [16], we adapt their naming convention with specific
changes to credit the original thoughts about tactile features. We
capitalize the first letter (which indicates the five major tactile
dimension from [14]: Friction F, Microscopic Roughness Mi,
Macroscopic Roughness Ma, Compliance C, Thermal Conduc-
tivity T), and add TUM as superscript. For example, our feature
addressing the SynTouch-inspired dimension of compliance
cCP is denoted as CCPTUM .
B. Material Scanners
Kuchenbecker et al. [17] defined the term haptography,
which describes the process of recording and displaying all hap-
tic properties of an object similar to capturing and showing an
image in photography. Unlike to the conditions faced in photog-
raphy, haptography data acquisition is complicated by the
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
necessary physical interaction with an object, and hence, is
influenced by active scan-time parameters. For example,
Romano et al. [18] have shown that acceleration signals result-
ing from tool–surface interaction heavily depend on the scan-
ning speed and force. Current technical systems that attempt
to solve this challenging task mainly concentrate on the
acquisition and display of vibro-tactile signals for roughness
acquisition and display using acceleration sensors during tool-
mediated material interactions as investigated by Kuchenbecker
et al. [17] and Culbertson et al. [19], [20]. These vibro-tactile
signals are also used to recognize material surfaces using
robots [21]–[25] and during human freehand movements by
Burka et al. [26] and in our previous work [27], [28]. Beyond
using tool-mediated accelerometer scanning setups, other meth-
odologies measure the vibro-tactile signal propagation through
the human skin as shown by Sano et al. [29], Vardar et al. [30]
and Visell et al. [31], or, are based on tribometry scans as for
instance conducted by Colgate et al. [32]. The vibro-tactile
sensing capabilities can even exceed human performance of
material identification, as shown by Fishel et al. [22] using the
BioTac sensor.
Acquiring and modeling other relevant tactile dimensions
has also been studied extensively in the past. For example, ste-
reoscopy-based approaches which determine the intensity of
surface structures have been presented recently by Choi
et al. [33]. Comparably, Dulik et al. [34] use infrared light
during non-contact scans to infer the surface height profile.
Thermal conductivities can be measured using thermistors or,
as shown by Aujeszky et al. [35] or Choi et al. [36], using a
thermal camera based on nondestructive infrared recordings.
Low-frequency hardness is usually modeled as spring stiffness
according to Hooke’s law as done by Basdogan et al. [37] and
can be measured using normal force measurements that are
divided by the indentation depth. High-frequency hardness is
related to tapping responses measured with accelerometers
and has been extensively examined by Okamura et al. [38]
and Kuchenbecker et al. [39].
Two different approaches exist to capture all the aforemen-
tioned tactile dimensions. A material scanner can be wielded
either by a human operator, or by a robotic scanning system
which precisely controls its interaction parameters.
A human operator-based approach reveals a specific vari-
ability in these scan-time parameters. For example, a sensor-
ized tool generally is moved within a speed range of about
20 mm/s up to 240 mm/s, as shown by Culbertson et al. [19].
The application of different sensing domains and the design of
robust features as shown in our previous work [27], [28] can
mitigate the dependencies of these parameters.
Robotic scanning setups, as presented by Sinapov et al. [40],
Jamali et al. [21], Fishel et al. [22] or Hoelscher et al. [24] have
the advantage of precise and controllable movement procedures
on predefined material sample slices. The current state-of-the-
art scanning platform is the BioTac Toccare [15], which allows
for a comprehensive scan of a predefined material sample to
assess its tactile properties and can be considered as the gold
standard for automated planar tactile sensing. Beyond two-
dimensional surface scans, Chu et al. [41] have shown that a
PR2 robot equipped with two BioTac sensors successfully
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
405
extracts relevant haptic properties to perform the task of object
recognition. The robot applied a series of four of the six EPs
(except lifting and contour following) on each of the 60 objects
used in the experiments. They computed the mean and the maxi-
mum value of the low-frequency signals measured with the
BioTac and converted the recorded high-frequency signals into
a non-normalized energy spectral density (ESD). The total
energy of the ESD curve, the spectral centroid, the variance, the
skewness, and the kurtosis transform the ESD into single-valued
features. Kaboli et al. [25] extended this idea using a Shadow
Hand equipped with five BioTac sensors to perform classifica-
tion of 120 objects with an accuracy of 100%. However, it is not
clear which of the presented material samples belonged to one
and the same material class. The ambiguous class names, such
as carpets (textures 58–78), do not allow for a generalization of
the material identification. It is also not clear whether such basic
features are inherently descriptive enough to sufficiently repre-
sent the material samples in a haptic context.
We argue that both approaches have their advantages and pro-
vide solutions for different applications. For example, the Bio-
Tac Toccare might be a good choice to scan planar material
samples for industrial or commercial applications, whereas
hand-held operator systems perform well as mobile and versatile
haptic property scanners. Generally, robotic systems can reliably
reproduce desired recording procedures and generate more sta-
tionary material interaction signals on planar material samples.
By contrast, a human operator can easily adapt the scan proce-
dure to complex object geometries and freely scan any material
sample without prior preparation of the scanned object. Humans
can evaluate much faster where and how a scan should be per-
formed, especially for very complex object geometries. Sec-
ondly, robotic solutions require great effort to reduce the motor
vibrations which affect the accelerometer recordings during the
robot-surface interaction. The cost and effort to set up and run
such a system are significantly higher than a human-operated
scanner. This point might be leveraged by cost scalability in the
future. To the best of our knowledge, the BioTac Toccare system
currently costs about $3  105 and the hand-held operator-
wielded device that we propose in this article about $3  103 .
C. Contributions
We envision a mobile human-operated material scanner to
capture multi-modal haptic data that is associated to meaningful
labels of real world material samples. We present mathematical
formulations of tactile features which were conceptualized and
partly summarized by other experts in the field to allow for suc-
cessful material classification in combination with audio-visual
features and to provide parameters possibly used for tactile dis-
play. The following list summarizes the main contributions of
this paper.
 We present a taxonomy based on geological, biological,
and chemical naming conventions and provide a novel
database (LMT Haptic Material Database) consisting
of 184 material classes. The collected images and data
traces have a total file size of about 80 GB.
 We present a mobile material scanning setup
(Texplorer2, two units are shown in Fig. 1), which
406
Fig. 1. Two components of the proposed Texplorer2.
extracts relevant material properties. A video describing
the system is included in our website [42] and uploaded
as supplementary material.
 We mathematically define perceptual tactile features,
denoted as TacTUM features, which are inspired by the
biomimetic process features from Fishel et al. [15], [16].
 We compare the hand-crafted feature-based results with
the classification results obtained with a commonly
used deep neural network based on a large number of
64,400 [500 x 500] RGB texture images from our new
database using the 184 defined material labels.
This paper is organized as follows: In Section II, we
describe our substantially novel haptic database taxonomy,
and Section III provides an overview of our hardware setup.
Section IV presents the multi-modal feature calculation, and
Section V their performance in the desired task of material
classification and further compares their performance to sur-
face image-based deep learning-based (DL) classification. We
conclude the paper in Section VI.
II. DATABASE AND CATEGORIZATION
Publicly available material databases for haptics research
(e.g., from Culbertson et al. [43], Hassan et al. [44], or our prior
database from our website [42]), provide a great variety of dif-
ferent haptic data scans. However, these databases are based on
colloquial material names (e.g., stone tile) which can lead to
ambiguities across different industrial or commercial branches.
In addition, these databases typically contain between 100 and
200 differently shaped surface textures which often belong to
the same material classes, though. Consequently, we performed
an interdisciplinary survey on the chemical, biological, and
geological material names and defined the material class labels
according to these observations.
A. Taxonomy
Our naming convention works as follows. C denotes the
major material class, e.g., metals or woods. The identifier S con-
siders the material sub-class, e.g., hardwood or softwood. M
denotes a specific material, e.g., lead, tin, or aluminum on a fine
classification level. Many materials come along with different
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
Fig. 2. Granite is an igneous (S1 ) stone (C6 ) which is either polished (left:
P2 ) or otherwise physically processed (right: pointed P3 ). The difference in
the surface structure heavily influences its haptic feel.
processing shapes P . For example, wood can be veneered or
naturally cut from the corresponding tree; aluminum can con-
tain differently shaped holes or gratings, and sandpaper pos-
sesses different grit sizes. The applied processing procedures
can lead to completely different visual and tangible perception,
although the actual material is the same. Another identifier X
denotes class-specific attributes. For example, we need to take
the moisture of woods into account, which changes their density
or metals may contain more or less percentages of their compo-
nents, especially steel alloys. Table I shows an overview of the
taxonomy with respect to the classes and sub-classes. The more
extensive Table 6 in the appendix further shows the different
processing shapes and extra class properties. We have also
included a complete material name list on our website [42].
The taxonomy is mainly inferred from Weber et al. [45]
and [46]. The database currently consists of 88 wooden, 24
organic, 18 plastic, 4 ceramic, 4 glass, 28 stone, 12 metallic,
and 6 composite material classes, in total 184. The relevance
of this taxonomy is given in the following based on the exam-
ple C6 S1 M1 (granite) and illustrated in Fig. 2.
Considering a pure granite tile we either may face a pol-
ished P2 (left image in Fig. 2) or a pointed P3 (right image in
Fig. 2) sample, which leads to a completely different percep-
tion of the material surface as well as different scan results.
B. Discussion
Due to the large amount of woods available, we currently face
a class imbalance towards C1 which needs to be considered
whenever the classification system has to distinguish on a coarse
scale between, e.g., woods and other classes. However, fine level
material classification is not affected since we have the same
number of observations (instances) for each material class.
We actively record new material samples to extend the already
extensive database. Several materials, e.g., concrete or glass,
exist in dozens of different processing shapes and mixtures. We
introduced the identifiers P and X to take these material-specific
properties into account and will significantly increase the number
of differently processed materials in the database in future work.
Considering the number of different sensing domains and
required data to represent these different material appearances,
we still face an ongoing task. That is also why we suggest to fol-
low taxonomies like the one proposed in this article to unambigu-
ously connect haptic data traces to the exact material. Note that
the number of woods goes well beyond the 10,000, for example.
The number of various minerals, stones, or metal alloys and their
processing shapes is similarly excessive.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES 407

TABLE I
TAXONOMY OF THE CLASSES C AND SUB-CLASSES S. ALL MATERIALS M ARE INTEGRATED INTO THIS LABELING STRUCTURE

Fig. 3. Overview of the Texplorer2 units. The operator bag and laptop tray
are not depicted.

III. TEXPLORER2 MATERIAL SCANNING SETUP

We seek to capture and parametrize material samples of the
different material classes. Consequently, a material scanning
system is required to record relevant device-surface interac-
tion data. Varying scanning parameters, such as speed and
force for contact-based approaches, or distance, rotation and
illumination conditions for non-contact approaches complicate
robust data acquisition. Fig. 4. First Texplorer2 unit for the acquisition of microscopic and macro-
scopic roughness information. Sensing components are shown in red. The
images showing the exploratory procedures are adapted from [1].

A. Texplorer2 Overview
Based on the idea of a haptic camera (haptography) from
Kuchenbecker et al. [17], we see the necessity for an intuitive
and flexible low-cost operator approach for the measurement
of any material sample. We completely redesigned the
Texplorer device from our previous work [28] and added new
components. Inspired by the design of the Proton Pack by
Burka et al. [26], we embed the scanning system into a bag to
allow for mobile scanning using a laptop, a laptop supporter/
tray (Rocket Packs) and a power bank (Anker). A USB data
acquisition card (NI USB-6002), attached to a Windows 7 lap-
top Lenovo 80VR (i7-7700HQ CPU at 2,800 MHz, 16 GB
RAM) collects the multi-modal sensor data. In addition to the
lower cost and simple scanning procedure, we further identify
the advantage of possibly scanning materials of arbitrary
shape, geometry, and size. The operator performs multiple
scanning steps with four mobile devices, which we denote as
Texplorer2 units (TU1–4).
Fig. 3 shows the four main units of the Texplorer2 and the
following list summarizes the general operating principle and
the target haptic dimension of each unit.
 TU1 records acceleration and audio signals to calculate
microscopic roughness features as well as reflectance
data for macroscopic roughness definition. It is slid
over the material sample.
 TU2 records normal and tangential forces to describe the
surface friction, pressure and folding FSR information to
describe hardness, and thermal cooling data to describe
warmth. Beyond, it acts as metal and magnetic detector.
This unit is first placed on the material sample to collect
static touch information, and thereafter, used to press on
and squeeze the material sample.
 TU3 captures surface images, magnified surface
images, and images with additional illumination using a
smartphone (Apple iPhone 8) camera.
 TU4 measures the mass and the volume to estimate the
density of an object sample by placing the sample on
top of the scale and inside the measuring jug.
B. Texplorer2 Units and Data Processing
1) Texplorer2 Unit 1: Micro- and Macroscopic Rough-
ness: The first Texplorer2 unit is rolled by the operator over
the surface of an object as shown in Fig. 4. Three bearings
ensure that the influence of the applied force is mitigated and
the distance between the surface and the IR sensor is constant.
a) Vibro-Tactile Signals: We use a three-axis accelera-
tion sensor ADXL335 (Adafruit) with a range of
3g to
acquire vibro-tactile signals v, i.e., to capture microscopic
roughness information with a sampling rate of 3,000 Hz. We
use two different kinds of tool tips during separate scans: the
bare finger which propagates finger-surface interaction signals

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
408 IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020

to a held accelerometer, and a stainless steel tool tip mounted

directly at the accelerometer.
We capture bare finger recordings vf;t during tapping on
and vf;s during sliding over the material surface. The operator
holds the acceleration unit between the thumb and index finger
as shown in Fig. 4 (upper middle illustration). The index finger
is placed on the accelerometer and simultaneously touches the
material surface directly. Additionally, tool tip tapping vt;t
and sliding vt;s recordings were collected after the bare-finger
recordings. The operator mounts a 3D-printed stainless steel
tool tip (hemispherical tip with a diameter of 6 mm) on the
ADXL335, clips it (using Velcro tape) on the TU1, and
repeats the same scanning procedure. The majority of the
operator force is compensated by the bearings during a scan.
We collect the two versions (bare finger-based and steel tool
tip-based) of the vibro-tactile signal recording procedures for
two reasons. We need to determine which works more robustly
for material classification, and, which signals are more appro-
priate for haptic display. During our testing we observed that
stainless steel tool tip recordings work better for classification,
presumably because they amplify the tool-surface interaction.
The bare finger recordings intuitively work better for haptic
display, which still needs to be proven in future work address-
ing the display of the 15 TacTUM dimensions.

b) Audio Signals: We place an omni-directional capaci-

tive microphone (Blusmart Lavalier Clip) on the TU1 to capture
audio signals of a frequency range between 10 Hz – 7,500 Hz
using a sampling rate of 44,100 Hz both for bare finger af;s as
well as for the stainless steel tool tip at;s audio signals, which
are recorded in parallel to the vibro-tactile signals.

c) Reflectance Signals: Dulik et al. [34] worked

with infrared reflective (IR) scans to extract height-related
material properties. We employ a comparable IR sensor
(Adafruit AF2349) consisting of an emitter, a detector, and a
10kV resistor in a voltage divider circuit. We measure one-
dimensional reflectance signals r to characterize macro-
scopic roughness information. Note that the IR sensor has a
fixed distance of 7 mm to the surface which is ensured due
to the stable rolling of the three bearings during a slide.
For completely flat surfaces, this distance is below the sens-
ing range, and hence, the values for flat surfaces are constant.
However, if a material surface contains coarse structures or
holes, the measured distance varies accordingly.
d) Data Processing: A raw vibro-tactile scan with TU1
is 5 seconds long. When the recording begins, only accelerom-
eter noise is present until the operator starts moving the TU1.
We shift a 0.1 ms frame over the raw vibro-tactile signal to
identify the start of the sliding motion. Compared to the signal
noise energy, the window frame energy rises notably during
this procedure which leads to the start index. We then extract
one-second-long acceleration signals as vibro-tactile data. The
audio data indices are the same due to the parallel recording
procedure. The vibro-tactile signals are band-limited between
10 Hz and 1,000 Hz to remove the effects of human hand
motion and accelerometer sensor noise. We combine the three
Fig. 5. Second Texplorer2 unit for the acquisition of thermal, friction, and
hardness material information. Sensing components are shown in red. The
images showing the exploratory procedures are adapted from Lederman
et al. [1].
acceleration signal components into one using the DFT321
algorithm from Landin et al. [47] as used by Culbertson et al.
in [48] and all following works.
2) Texplorer2 Unit 2: Warmth, Friction, and Hardness:
a) Thermal Signals: Thermal conductivity properties
help to distinguish between different object surfaces [1].
Although surfaces conduct heat from the human skin differ-
ently during bare-finger touch, they obtain ambient tempera-
ture after an indefinite time. Aujeszky et al. [35] use a laser to
heat a material sample and locally measure thermal cooling
using infrared sensing. We follow a comparable approach to
infer thermal properties of the material samples. Our setup is
shown in the upper part of Fig. 5. We actively heat the mate-
rial surface and measure the subsequent local cooling of the
material surface. More specifically, we heat a Peltier element
(TES1-127021) for three seconds while pressing it on the
material surface. Subsequently, we move the unit to measure
the thermal cooling rate over the heated spot using an infrared
temperature sensor (Melexis MLX90614) for ten seconds and

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
Fig. 6. Close-up view of the static touch part of TU2. The electric conductive
tissues reliably detect metals, the Peltier element and MLX90614 measure
thermal cooling, and an additional magnetometer unit (right) instantly detects
the change of the external magnetic field caused by the neodymium stack
through a material.
denote the recorded temperature data array as t. Note that the
required change in position can be automated in a future ver-
sion of the Texplorer2.
b) Force Sensitive Resistor (FSR) Signals: Surface
friction information is recorded using the index finger during
a sliding motion of the object surface. Our second Texplorer2
unit contains two FSR (FSR400, Interlink) for normal and tan-
gential sliding force measurements, see Fig. 5, middle. These
sensors are used to determine the exerted normal force and the
tangentially exerted friction force during the sliding move-
ment. In the following, we consider the intensity values of
these sensor measurements only; the orientation of the forces
is captured by the sensor placement (normal and tangential).
We denote the force arrays f ¼ ½f1 ; f2 ; . . .fn  from a scan as
f0fr;n and f0fr;t , respectively. The raw FSR sensor values f0 (in
volts) grow logarithmically with increasing force. We convert
each voltage value f 0 in f0 to a force applying the expression
0 1
f ¼ 9.81  m=s2  0.01 kg  ef 1:33 V (1)
according to the inverse force-voltage relation diagram (Inter-
link FSR integration guide), leading to the normal and friction
force arrays ffr;n and ffr;t . Note that the normal FSR, which
contacts the surface, is equipped with a 6 mm diameter PLA
sphere.
The same FSR used for the normal force is also used to infer
hardness properties. Additionally, a Flex Sensor (FS-L-0095-
103-ST, Spectra Symbol) collects information about the object
flexibility. For example, if a piece of paper lies flat on a table
and a robotic material scanner presses on it, the hardness of the
underlying table is measured. However, humans can easily lift
the paper to fold it. In this case, the Flex Sensor will change its
angular value indicating flexibility. Note that the FSR for the
normal force from the friction estimate is used alongside with
the Flex Sensor during the pressing and folding procedure. This
step helps to distinguish between papers, leather, and fabrics,
for example. The operator first presses on the material surface
and releases it, using the FSR for normal force acquisition. Sec-
ondly, the operator grabs the whole material, if possible, and
tries to fold it between the thumb and index finger. The Flex
Sensor records the folding angle a f while the normal force FSR
records the applied folding force ff in parallel.
c) Metal and Magnet Detector Signals: As in our previ-
ous work [28], we use two electric conductive tissues as elec-
trodes (see Fig. 6, left) during the static contact with a
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
409
Fig. 7. Recorded magnetic data during static touch. Aluminum (left) does
not change the magnetic field, steel wool (middle) is paramagnetic, and cast
iron (right) reveals significant magnetic properties (ferromagnetic). Diamag-
netic materials, which are currently not part of the database, would generate
positive values. All other materials lead to a mean value of DB ¼ 0.
material to identify metals. This procedure leads to an inverted
binary array m ~ 2 f5 V, 0 Vg, which we map to the values
m 2 f1; 1g. Only electrically conductive materials, e.g.,
metals, pull down the DAQ input to ground (0 V) during scan
time, leading to an m ¼ 1 array. We set a binary value
metal ¼ 1 if m > 0.
As a minor contribution, we add a magnetometer to measure
the change of the magnetic field through a material sample.
More specifically, we apply a stationary external magnetic field
in a constant distance (thickness of casing) and measure the
influence of the underlying material sample. We 3D-printed a
casing which contains a 3-axis digital magnetometer
MAG3110 (Sparkfun) PCB and a stack of eight neodymium
permanent magnets, shown in Fig. 6 (right). We zero-calibrate
the MAG3110 using these permanent magnets which expose a
strong and constant magnetic field to rule out the influence of
the Earth’s magnetic field. The resulting data traces are denoted
as DB and shown for selected materials in Fig. 7.
Magnetism is not a tactile property, but the change of the
magnetic field characterizes several metals and their alloys.
We can reliably distinguish between ferro-, non-, and diamag-
netic materials within a second during static touch and thereby
even exceed human material classification capabilities. The
thickness of the material sample ideally should be included in
the future, still. Features based on metal or magnetic material
detection as well as thermal cooling are not influenced by
other sensing domains (such as vibro-tactile signals) and lead
to further independent, and hence, characteristic features.
d) Data Processing: Since the operator index finger
may create an offset force while touching the TU2 friction
FSR, we need to subtract the mean value of the first force sam-
ples from the friction force signal.
3) Texplorer2 Unit 3: Visual Properties: Visual informa-
tion is linked to haptic perception as summarized by Lacey
et al. [2], and hence, contains information about relevant
material properties.
a) Camera: The images of the materials, denoted as
Idisp , are captured using an iPhone 8 (Apple) camera with a
resolution of 3024 x 4032 pixels. To infer macroscopic details,
we further attach a magnifying lens (four times zoom) and
capture macro images Imacro . Note that the distance between
each surface and the camera was held constant using a Styro-
foam bumper to ensure the same recording distance and per-
spective. We further use the camera flashlight to capture
410
Fig. 8. Volumetric approach based on Archimedes’ principle. We use
different measuring jugs to infer the volume of arbitrarily-shaped objects by
displacing either water or micropearls (diameter 1 mm). The images showing
the exploratory procedures originate from Lederman et al. [1].
illuminated images Iillu which provide further information
about the object’s surface reflectivity.
b) Data Processing: We remove 100 pixels from each
border of the image to reduce the number of artifacts.
4) Texplorer2 Unit 4: Mass and Volume: The material
samples are placed on an electronic scale (Smart Weight,
range of 0.1 g – 2 kg) to measure their weight m which is
noted manually. The density r ¼ m V has been determined and
published online for the majority of all available materials
(e.g., in [49]). However, for a new query to the database, very
complex object geometries, and undocumented densities, we
still need an approach to estimate the volume V . This task is
complex and an automatic robotic setup may not be able to
extract the volume of complex object geometries without
additional effort, time, and cost, whereas a human operator
can intuitively carry out displacement techniques to estimate
the volume. For example, a set of measuring jugs can be filled
with water and the object is placed inside. The displacement is
proportional to the volume of the object. If the object could be
damaged by water, we use polystyrene micro-pearls instead.
To read the measuring scale accurately and avoid uneven
micro-pearl distribution, we place a vibration motor (Newgen
Medicals Vibration Plate) below the jugs and apply a constant
20 Hz vibration before each manual reading of the scale as
shown in Fig. 8. The overall displacement procedure has
major advantages compared to infrared or stereoscopic
approaches. First, the volume of concave objects or hole-con-
taining ones can be determined, and second, the volume of
highly reflective objects can be measured.
C. Recordings
The TUM Department of Architecture holds the largest col-
lection of building materials in all German universities [45]. We
scanned most of the available material samples along with mate-
rials from our previous work [28] to set up the largest public hap-
tic database (to the best of our knowledge). In our current version
of the database, two human operators scanned each material
(five times in total) and stored the data traces as. mat files which
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
are available on our website [42]. This collection is intended to
contain a broad set of different recording modalities. The data-
base shall allow other researchers to test various material classi-
fication approaches and define further relevant tactile features.
Available material archives, e.g., the one presented in [50],
indicate which materials still need to be added in the future.
D. Discussion
We do not explicitly measure the scanning parameters during
the device – surface interaction. Burka et al. [51] have shown
that variable scan-time parameters do not necessarily decrease
the classification performance whenever options exist to miti-
gate their influence by intelligent acquisition device design or
the application of robust features. Several aspects of the
Texplorer2 account for these dependencies, and we also identify
more options for future optimization. The bearings of TU1 com-
pensate for heavy pressure forces during the acquisition of
vibro-tactile signals. The fixed distance of the IR reflectance sen-
sor to the surface also makes sliding scans robust in terms of sur-
face structure detection. The major remaining dependency is the
scanning speed. Different operators may scan the surfaces differ-
ently fast, yet within a specific range of this dependency. Cul-
bertson et al.’s [19] work on tool-mediated setups shows an
average speed of 130 mm/s which gives a good indication of
how fast humans wield such acquisition systems. We need to
accept a remaining variance in the vibro-tactile and audio signal
scans, but are confident that the inclusion of other modalities,
good device design, and robust features can still lead to success-
ful classification.
Another important point to discuss is the number of modalities
used in this article. Fig. 9 shows all currently recorded modalities
for one material sample. Haptic material interaction is a multi-
faceted procedure; we already employ nine different sensors
leading to twenty different kinds of signals. The application of
different sensors reduces potential correlations from subse-
quently extracted features, but poses challenges for the required
amount of data to be collected. We believe that content-based
haptic data acquisition, potentially used for display as well,
requires multi-modal data acquisition. Versatile material scan-
ners are required to cope with this challenge and potentially will
add further sensors. For example, another sensing domain not
yet examined on larger material databases is the electric suscep-
tibility x, which is very characteristic for electrically non-con-
ductive materials.
IV. FEATURES FOR MATERIAL CLASSIFICATION
We define and use a set of tactile as well as non-tactile fea-
tures based on the multi-modal data traces introduced in the
following section. Note that several parameters of the features
are the result of feature optimization and were iteratively
determined.
A. TacTUM Tactile Features
Fishel et al. [16] propose a set of representative tactile fea-
tures which we understand as a more detailed description of
the five major tactile dimensions from Okamoto et al. [14]. To
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
Fig. 9. Example data traces of one material entry (polyurethane). Overall,
nine sensors record twenty different data traces, which are the source for our
features.
the best of our knowledge, there is no explicit mathematical
definition of these features publicly available. Hence, and as a
major contribution, we propose the equations for the calcula-
tion of these features in the following.
1) Macroscopic Roughness: The macrotexture (MaTXTUM ),
macrotexture coarseness (MaCOTUM ), and macrotexture regu-
larity (MaRGTUM ) describe the macroscopic roughness features
of a material surface.
a) Macrotexture MaTXTUM: We define a surface-
dependent macrotexture feature based on our macroscopic
roughness strength feature from [28]. Several sensing
dimensions determine the physical existence of surface
structural patterns. We believe three different signals to be
appropriate for calculating this feature value. Table II shows
the auxiliary functions which we use for the MaTXTUM
calculation.
The first component is related to acceleration spikes in the
vibro-tactile signals. We define a spike detection algorithm
on the absolute values of the vibro-tactile sliding signals
with N being the length of vabs . We first compute a 100-
ð100Þ
point window simple moving average array vabs of vabs .
The moving average window length is an optimization
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
411
TABLE II
MACROTEXTURE NOTATION OVERVIEW
parameter which can be made adaptive to the speed in future
work. We add the standard deviation sðvabs Þ and the mean
of vabs to obtain an adjusted array to compensate amplitude
variations for different recordings of one material sample
and calculate
ð100Þ
vth ¼ ðsðvabs Þ þ vabs Þ  1 þ vabs (2)
vabs;D ¼ vabs  uðvabs  vth Þ (3)
which is computed using the unit step function uðxÞ that
returns 1 element-wise if the corresponding array entry is
greater than 0, and 0 otherwise. We then define the vibro-tac-
tile spikiness (VS) component as
VS ¼ log 10 ð1 þ vabs;D Þ (4)
We further consider friction signal variations (FV) during a
slide as another component for MaTXTUM . Rapid changes in
the friction force signal ffr;t indicate a change in the material’s
underlying physical structure, e.g., while sliding over a coarse
or bumpy surface profile.
ð100Þ
FV ¼ sðffr;t  ffr;t Þ (5)
As a third component, the IR sensor’s reflectance signal
data is related to the height profile of a material surface. Dur-
ing a scan, variations in the reflectance signals capture the
change of the current surface-sensor distance. We calculate
the moving average rð100Þ and consider the standard deviation
of the difference r  rð100Þ as our reflectance variation (RV)
value
RV ¼ sðr  rð100Þ Þ (6)
Finally, MaTXTUM is calculated as a linear combination

1 MaTX TUM ¼ 1  VS þ 2  FV þ 3  RV (7)
using the weighting parameters 1;2;3 which are determined
during feature optimization (Section V-A2) as (3, 2, 4).
b) Macrotexture Coarseness MaCOTUM: The existence
of visual patterns on a surface indicates whether the material
surface is coarse or not. Consequently, we apply the Harris
Corner Detector (HCD) [52] to identify the n most character-
istic image corner points in our surface images Idisp . We use
the speeded-up robust feature (SURF) extractor [53] on these
corner pixels to extract and sort the scale property sc in a
descending order array sc (using the Matlab function
detectSURFFeaturesðÞ). We consider the visual
412 IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020

component of macrotexture coarseness as the mean value of

the n largest scale factors sc1...n identified in Idisp multiplied
by the spectral spread of the image histogram (introduced as
contrast CðÞ by Tamura et al. [54]). CðIdisp Þ is hence the
image contrast of the material images; low-contrast images
are more likely to be of completely flat material surfaces,
e.g., plastics or glasses. We then multiply the result by the
mean value of the absolute vibro-tactile signal which trans- Fig. 10. Image spectrum before (left) and after (right) applying the inverted
forms this image feature into a tactile feature. Otherwise, Gaussian smoothing filter.
painted structures on flat materials potentially distort the fea-
ture value. Hence, vibro-tactile signal energy and the mean value of all pixels
in Xfilt as

2 MaCOTUM ¼ vabs  CðIdisp Þ  sc1...n (8)
with n being 10 in this paper resulting from feature optimiza-   filt  CðIdisp ÞÞ
3 MaRGTUM ¼ vabs  ðX (12)
tion (Section V-A2).
c) Macrotexture Regularity MaRGTUM: Visible texture We subtract the image contrast for two reasons. First, homoge-
regularity is related to the intensity and arrangement of the neous images with low contrast are regular; secondly, high-
image spectral components. Regular structures reveal specific contrast irregular surface images should lead to a very small
frequency peaks, whereas irregular image spectra show an MaRGTUM value. The multiplication of vabs follows a similar
incoherent noisy pattern. The real part of the image spectrum motivation as for MaCOTUM ; we need to prevent that irregu-
! larly painted structures on actually flat (homogeneous) surfa-
1 X N1
M1 X
0
X ðk; lÞ ¼ Re pffiffiffiffiffiffiffiffiffiffiffiffiffi Iðm; nÞ  e2pjðmk
M þnl
N Þ
(9) ces distort the feature value.
M  N m¼0 n¼0
2) Microscopic Roughness: The microtexture roughness
is calculated to obtain a spectral representation which is then (MiROTUM ) and microtexture coarseness (MiCOTUM ) describe
centered using Xðk; lÞ ¼ X 0 ðk  M2 ; l  N2 Þ. Thereafter, a the microscopic roughness features of a material surface.
threshold t is defined
a) Microtexture Roughness MiROTUM: The wavelet
t ¼ a  |ffl{zffl}
max Xðk; lÞ
decomposition of a vibro-tactile signal vt;s (during a slide
1kK
1lL using the stainless steel tool tip recordings) leads to approxi-
mation and detail signals which change comparably for differ-
with a being 0.75. Consequently, all values smaller than 75% ent recording speeds. Our microscopic roughness feature from
of the maximum value are removed using [28] is based on the difference of two of such different detail
 levels by calculating
Xðk; lÞ if Xðk; lÞ t
X thresh ðk; lÞ ¼ (10)
0; otherwise. 
4 MiROTUM ¼ log 10 ð1 þ dÞ (13)

The remaining peaks represent the most characteristic spatial with

frequencies of the surface image. Irregular structures have
several peaks around the center of the real part of the image D1 ðvt;s Þ
spectrum, whereas regular images reveal repetitions up to d ¼ D1 ðvt;s Þ   D5 ðvt;s Þ: (14)
larger spatial frequencies. That is why an inverted [200 x 200] D ðvt;s Þ
5

pixel Gaussian smoothing filter G0 ðk; lÞs ¼ 1:0  Gðk; lÞs ¼

k2 þl2 Dk ðvÞ contains the absolute values of the kth detail level of v.
1:0  1
e
2ps 2
with s = 25 is applied on each element in
2s 2
The used wavelet type is a Daubechies 8 (db8) wavelet result-
the spectral domain of Xthresh to underweight closer peaks to ing from the feature optimization (Section V-A2).
the center using the Hadamard product b) Microtexture Coarseness MiCOTUM: We adapt the
spectral centroid definition from Fishel et al. [22] based on the
Xfilt ¼ Xthresh
G0s : (11) real part vibro-tactile signal spectra Vt;s
Fig. 10 visualizes this step for the image spectrum of a regu- PN
larly perforated epoxy plate (C3 S4 M2 P2 X1 ).
 i¼1 f i  jVt;s ðf i Þj
Spectra of irregular surface images scatter close to 5 MiCO TUM
¼ fs  PN (15)
i¼1 jVt;s ðf i Þj
the center, whereas regular surface images can reveal
specific peaks far from the center. The threshold t and the
filter dimensions are result of the feature optimization with fs ¼ 3000 Hz being the sample rate of the accelerometer.
(Section V-A2). Finally, the Macrotexture Regularity Smaller values indicate the presence of a fine (i.e., smooth)
(MaRGTUM ) is calculated as the product of the mean material surface, e.g., a polished steel surface.

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
Fig. 11. Separation of normal and tangential (friction) forces for the calcula-
tion of tactile stiction FSTTUM and sliding resistance FRSTUM . Note that the
increase of the friction force before the normal force increase may result from,
e.g., operator finger movements inside the TU2.
3) Friction: Tactile stiction (FSTTUM ) and sliding resis-
tance (FRSTUM ) can be considered equivalent to the static and
dynamic friction coefficient of a material surface.
a) Tactile Stiction FSTTUM and Sliding Resistance
TUM
FRS : As long as the normal force ffr;n is about zero, the
operator did not noticeably press on the material. An increas-
ing normal force then indicates the beginning of the static fric-
tion phase. Normal and friction force ffr;t values are then used
to model friction based on Coulomb’s law.
Since we can not directly measure the transition from stick-
ing to sliding over the surface with TU2, the ending segment
of the static phase needs to be determined following another
approach. We observe that the friction and normal forces
increase differently whenever the operator starts touching the
material in order to perform a slide. We shift a 10 ms window
over the signals to identify the maximum positive slopes
Dffr;max and Dfn;max and define tactile stiction as
Dffr;max

6 FST TUM ¼ (16)
Dfn;max
Fig. 11 visualizes exemplary signal traces and the correspond-
ing slopes.
The following signals after the ending of the static segment
describe the sliding phase until the normal force again
decreases to zero. The FSR signals in the sliding segment are
denoted as ffr;FRS and fn;FRS , respectively. Consequently, slid-
ing resistance is calculated as
ffr;FRS

7 FRS TUM ¼ (17)
fn;FRS
b) Adhesive Tack ATKTUM: The effort required to break
contact with a surface [15] is proportional to the adhesive
forces of a surface. We define it as a quintary operator-decided
value
8
>
> 0 if not adhesive
>
> fn;max ¼ max f n;i
< 0:25 if slightly adhesive

8 ATK TUM ¼ 0:5 if uncertain (18)
>
> being the maximum force and D x = 4.0 mm measured with
>
> 0:75 if adhesive
: a caliper. If the material sample is deformed up to this inde-
1 if strongly adhesivecases
There are several reasons not to define this feature based
on measurements from the sensing device. First, the major-
ity of naturally occurring material surfaces are not
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
413
Fig. 12. Texplorer Unit 2 while being pressed on a deformable material sam-
ple. The normal force FSR (in red color) is used to infer the tactile compliance.
Note that the indentation depth is constant (thanks to the attached hemisphere
and TU2 casing) after the operator presses D x ¼ 4.0 mm into the deformable
material sample.
noticeably adhesive. Secondly, the adhesion properties of
any sensing device also differ from those of a human fin-
ger; a material surface scanned with a biomimetic sensor
made of silicone may result in a larger adhesive tack fea-
ture value than expected. Thirdly, an operator can intui-
tively assess the stickiness of a material during several
trials. Next, any oily or sticky surface might damage the
device, and the detection of such surface conditions is still
challenging for robotic systems. Lastly, the inclusion of
the operator judgment represents another variable not
depending on any other sensing domain.
4) Compliance: Operator-based material scanning
approaches reveal an advantage over robot-based
approaches for complex object structures in terms of hard-
ness estimation. Humans generally identify the most suited
locations for grasping and folding/pinching a material sam-
ple. We use the recorded normal FSR and Flex sensor sig-
nals during the pressing and folding actions of the operator
to infer hardness-related features. We first determine if a
material is deformable or not with regard to reasonable
human interaction forces. The folding angle values a f of
the TU2 act as tie-breaker in this binary decision. We
evaluate the expression sða af Þ < 1o prior to any compli-
ance feature calculation. If an object is rigid, all five com-
pliance features are set to zero.
a) Tactile Compliance CCPTUM: Tactile compliance
refers to the spring stiffness [37] of a material sample and is
defined as the ratio of the required normal force fn and
achieved indentation depth Dx. Thanks to the design of our
Texplorer2, we are faced with a fixed Dx and measure the cor-
responding (normal) force as shown in Fig. 12.
The tactile compliance then is

0 a f Þ < 1o
if sða

9 CCP TUM ¼ fn;max (19)
Dx else
with
1in
(20)
ntation depth, the additional force is compensated by the
whole TU2 casing. Note that a 3D printed PLA or rubber
(Shore hardness scale value > 75) hemisphere are applicable
in this context.
414 IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020

b) Local Deformation CDFTUM: We calculate the local

deformation around the measuring device as


10 CDF TUM ¼ sða
af Þ (21)

which describes how the object can be folded during free

interaction based on the achieved folding angle a f . Fig. 13. Texplorer Unit 2 while pinching a deformable material sample. The
normal force FSR and the Flex Sensor are shown in red color. Damping and
relaxation can be defined by the ratio Dff and vf ¼ Da
af =Dt.
c) Damping CDPTUM and Relaxation CRXTUM: Damp-
ing can be understood as the ratio of the counteracting force
and the current speed (Hartog et al. [55]). We measure the material sample to identify yielding correctly, which a human
temporal change of the folding angle array a f values and the operator can anticipate thanks to human knowledge about
required folding force ff;max . Note that these force values still materials. We hence redefine yielding as a quintary value
result from the normal force measurements, but are denoted 8
with subscript f to indicate the folding phase. Fig. 13 illus- >
> 0 if material recovers shape
>
>
trates how the TU2 implements this definition. < 0:25 if material likely recovers shape
The damping phase identification starts whenever the normal 
13 CYDTUM ¼ 0:5 if uncertain
>
>
>
> 0:75 if material unlikely recovers shape
force increases and ends at the maximum force value. This :
1 if material stays deformed
index range is defined as Dtdamp ¼ f1s  ðnmax;damp  nmin;damp Þ
with fs ¼ 3000 Hz being the sample rate of the DAQ. We (24)
extract the corresponding force difference ff;Dtdamp ¼ f f;nmax 
to account for the state uncertainty for such materials after
f f;nmin;1 which leads to
operator interactions. Only reasonable forces are considered
(
0 a f Þ < 1o
if sða during common human–surface interactions.

11 CDP TUM ¼ ff;Dtdamp (22) 5) Warmth: Observations from Bergmann Tiest et al. [56]
v else show that we can assume a heat dispersion model for material
f
Da surfaces. According to Newton’s continuous law of cooling,
with vf ¼ Dt f .
damp heated material surfaces cool down following the expression
We interpret tactile relaxation as the difference of the maxi-
mum occurring force and the sustained force value, i.e., the dT ðtÞ
last value fn;end of the force recordings using the normal force  ¼ r  ðT ðtÞ  Tamb Þ (25)
dt
FSR during pressing on the material sample.
with Tamb being the ambient and T ðtÞ the current temperature.

12 CRX TUM ¼ fn;max  fn;end (23) This differential equation can be solved to

T ðtÞ ¼ Tamb þ ðTmax  Tamb Þ  ert (26)

This definition is also only applied if the material sample is
deformable at all. Additionally, the majority of material sam-
assuming that surfaces cool down to ambient temperature after
ples sustain their maximum force. However, several samples
an indefinite amount of time. Note that r is an object-specific
slightly deform after the application of force and sustain only
constant containing the mass m, the touched area A, the ther-
a fraction of the maximum force value until the end of the
mal conductivity  and the thermal capacity c
recording.
A
d) Yielding CYDTUM: Yielding is defined as the degree r¼ (27)
to which a surface remains deformed after being pressed [15]. cm
We let the human operator identify this feature value without
using our Texplorer2 device. The motivation is similar to the and is, consequently, related to the object’s thermal cooling
adhesive tack feature definition. First, the majority of the after initial heating. We use our Texplorer unit 2 from Section
scanned materials are not deformable at all, and even if III to heat up the material surface and, subsequently, measure
deformable they completely recover to their initial shape. A the thermal cooling.
human operator can quickly assess this fact whereas a robot- a) Thermal Cooling TCOTUM: Our discrete thermal sig-
based system requires more time and possibly introduces nals t are sampled at fs ¼ 20Hz. Using
unnecessary scanning uncertainty. For example, a piece of
paper placed on a table is identified as rigid by a planar-work- tmax ¼ max1in ti (28)
ing robotic system, but the operator intuitively lifts it, recog-
and measuring, e.g., tamb ¼ 19
C during our recordings, we
nizes that it is not rigid, and also identifies that it might stay
infer the first thermal feature TCOTUM as
deformed after interaction depending on the applied force.
Additionally, a robotic system as well as a purely sensorized
manual approach needs to irreversibly change or destroy the 
14 TCOTUM ¼ tmax  tamb (29)

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
Insulating materials locally heat up to a significant tempera-
ture, whereas metals have almost no measurable difference
compared to their initial surface temperature. Note that the
ambient temperature needs to be recorded whenever a new
recording session starts.
b) Thermal Persistence TPRTUM: The calculation of
TUM
TPR needs to consider two relevant aspects. First, the
maximum temperature after five seconds (proposed by [16])
TCOTUM ¼ tmax;5 s  tamb and the temperature change at the
end of the recording tend  tamb indicate the thermal behavior.
Secondly, we square both terms to emphasize the larger abso-
lute temperature differences (compared to ambient/room tem-
perature) leading to
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

15 TPRTUM ¼ ðtmax;5 s  tamb Þ2 þ ðtend  tamb Þ2 (30)
of our recorded thermal signals t. A perfect insulator with sig-
nificant thermal persistence causes a large initial temperature
increase tmax and maintains a large value tend after a specific
time as indicated by Fishel et al. [16], e.g., up to 10 seconds in
our recordings.
B. Tactile Features from Related Work
Other researchers in the field have proposed suitable fea-
tures based on signal statistics. For example, Chu et al. [41]
have shown that the standard deviation and kurtosis of
recorded vibro-tactile signals are characteristic features, or
Fishel et al. [22] used features such as the average signal
energy and spectral centroid to achieve a remarkable classifi-
cation accuracy of 95.4% on 117 textures.
C. Hand-Crafted Image Features
Liu et al. give a comprehensive overview of the last
two decades of image-based texture classification in [57].
The description of all presented approaches goes beyond the
scope of our article. The main statement of [57] is that hand-
crafted feature approaches as well as convolutional neural net-
work (CNN)-based approaches enable a robust classification
of the existing image texture databases. However, these data-
bases only contain a small set of all available materials
and their processing shapes. Based on our interdisciplinary
naming convention, we consider our database with currently
184 material labels as an appropriate candidate for further
evaluation of image texture features for both hand-crafted
and deep learning-based approaches consequently.
Surface images reveal significant material information for
classes like wood or stones. Consequently, we adapt the most
promising surface texture feature definitions from Haralick
et al. [58], Tamura et al. [54], and use the corresponding
Matlab implementations from Monzel et al. [59] and Sdhir
et al. [60]. We further work with features like the dominant
color (Icol ) or edginess (Iedg ) from our work in [27] and calcu-
late ten rotation-invariant local binary pattern (RLBP) descrip-
tor values proposed by Ojala et al. [61]. In total, we calculate
31 surface image features on Idisp , Imacro , and Iillu respectively,
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
415
TABLE III
OTHER FEATURES OVERVIEW
leading to 93 hand-crafted image features. The following list
shows which features are used:
 Color (Icol ), brightness (Ibr ), edginess (Iedg ), and aver-
age values of 80%, and 85% of the largest peaks in the
real part of the normalized DFT2 (Idft1;2 )
 14 Haralick [58] (e.g., sum of entropy Ientr and image
complexity Icom ) and two Tamura [54] features: coarse-
ness (Icoa ) and contrast (Icon )
 Ten rotational-invariant local binary pattern (RLBP) [61]
D. Hand-Crafted Audio Features
We use the Matlab implementation from Giannakopoulos
et al. [62] to calculate a set of 35 well-known audio features
like the spectral centroid, zero crossing rate, and the Mel fre-
quency cepstral coefficients. Note that we apply these on all
finger and stainless steel tool tip audio signals af;s and at;s dur-
ing sliding, respectively, leading to 70 audio features.
 Zero-crossing rate, mean signal energy, signal entropy
 Spectral centroid, - spread, - roll-off, - entropy, - flux
 Harmonic ratio (HR) and fundamental period (F0)
 13 MFCC
 12 Chroma values
E. Other Features
Table III shows a set of common signal statistics and non-
tactile features which do not fall into any of the previously
mentioned feature domains, but reveal additional and indepen-
dent material classification abilities.
F. Discussion
The materials in the current database do not vary much
along the two tactile dimensions CYDTUM (yielding) and
ATK TUM (adhesive tack); almost all material samples retain
their initial shape and have no adhesive surface. However,
these features are completely independent from the other pro-
posed features thanks to their different sensing modality and
potentially are relevant for haptic display of adhesive surfaces
and sustained surface deformation. Similarly, relaxation
CRX TUM has a limited descriptive performance on the current
database. Most of the samples are either rigid, or whenever
compliant, naturally maintain the maximum force while
actively being pressed. However, if further materials were
added to the database, these features potentially enable a more
subtle classification.
The features not falling in the audio, image, or tactile
domain are very characteristic for specific material subclasses.
Notably, the density is the best feature to discriminate between
the variety of woods, or, the combination of the metal,
416
Fig. 14. Example boxplot of feature CCPTUM of class C2 (biodegradables
like furs or leather). The feature value is always set to zero for stiff materials
(if sðaaf Þ < 1o ), whereas it has a good distinction performance for soft
materials.
dominant color, and magnetic feature works well for metal
classification. The surface reflectivity is also highly character-
istic and independent of operator exploratory motions since
the data is captured during static touch.
V. SUPERVISED MATERIAL CLASSIFICATION
The primary goal of this work is the classification of the dif-
ferent material samples from the database recordings.
A. Data Preparation
This section shows how the features are evaluated and used
to set up the feature space before we train a classifier.
1) Feature Quality and Correlation: The most important
ingredients of machine learning are content-based, descriptive,
and independent features [63]. Our features are heavily
inspired by the major tactile dimensions from Okamoto
et al. [14] and Fishel et al. [16], and hence, we believe that
they fulfill the first condition.
Since the individual material scans are assumed to be inde-
pendent, one-way analysis of variance (ANOVA) is conducted
on each feature to assess its descriptive quality based on the
Kruskal-Wallis test. Boxplots visualize the feature values, as
shown for CCPTUM and a subset of materials in Fig. 14.
Post-hoc analysis based on the ANOVA statistics performs
pair-wise comparisons between all class means (Matlab mult-
compare implementation, alpha = 5% and Tukey-Kramer pro-
cedure for balanced one-way ANOVA with equal sample
size). Larger p-values indicate low discrimination between
two classes, and hence, we define the feature quality (FQ) as
FQ ¼ 100%  ð1  pÞ
based on the average of all p-values p resulting from the pair-
wise comparisons NðN 1Þ with N ¼ 184 being the number of
2 values between 0 and 1) feature space X01 . Each feature vector
material classes used in this article. A descriptive feature
shows a small intra-class and a large inter-class variance,
and hence, has a large FQ value. Note that this metric is
also used as one target parameter for feature optimization in
Section V-A2.
Thirdly, the features ideally are as weakly correlated as pos-
sible to guarantee feature independence. We consider the
Spearman rank correlation rspearman between two features as
meaningful metric. Note that the Pearson’s correlation coeffi-
cient is less appropriate for this task due to the potential
non-linearity in the data [64]. Whenever the absolute value of
correlation indicates stronger correlation, i.e., rspearman 0.5,
one of the two compared features should be excluded from the
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
Fig. 15. Explained variance over the principal components.
feature space. Larger correlations can be expected for sub-
dimensions belonging to the same major tactile dimension.
For example, FSTTUM and FRSTUM belong to the dimension
of friction and are intuitively correlated.
2) Feature Optimization: Several features from Section IV
contain parameters that need to be optimized in terms of fea-
ture quality. For example, the optimal wavelet type for the fea-
ture MiROTUM needs to be determined by testing different
wavelet types followed by an evaluation of how the feature
quality and its correlation to the other features changes. Our
optimization approach follows the same idea of high feature
quality (FQ) and low correlation between the features. The
corresponding feature space columns need to be recalculated
using different parameter sets. We ran a grid search of poten-
tial parameters to identify the maximum FQ and minimum
correlation values. For example, the weighting parameters
1...3 of MaTXTUM result in a set of (3, 2, 4) indicating that the
influence of the reflectance sensor data is most relevant and
also causally independent since the reflectance sensor is not
used for any other TacTUM feature.
3) Raw Feature Space Creation: Our initial feature space
X0 is composed of 193 features (columns f of X0 ) before fea-
ture selection takes place. The last additional five columns are
reserved for the responses y (true labels) based on our naming
convention Cx Sx Mx Px Xx . For example, coarse classifica-
tion yC considers the eight major classes C1 – C8 regardless
of the sub-class or material identifier. Five observations per
material compose the instances (rows) of X0 . We split our fea-
ture space based on k-fold cross-validation (five folds) using
the function cvpartition() from Matlab.
(31)
4) Feature Space Scaling: In order to make the features
comparable to each other [63], we create a normalized (feature
f i fmin
entry, i.e. feature fi , is normalized by applying f 0i ¼ fmax fmin .
We denote the raw feature space X01 as X in the following.
5) Feature Space Reduction and Selection: It is general
practice to keep the number of features as low as possible to
avoid over-fitting while using a sufficient number of descriptive
features to avoid under-fitting as highlighted by Domingo
et al. [65]. One approach to reduce the initial feature space is
dimensionality reduction. We first performed a principal compo-
nent analysis (PCA) on the initial feature space to assess if signif-
icant linearly uncorrelated variables could be identified.
However, the three major principal components only explain
about 30% of the variance, and the first ten about 62%, as shown
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
TABLE IV
BEST FEATURES FROM DIFFERENT TESTED DOMAINS
in Fig. 15, indicating that a large number of principal components
would be required to allow for successful material classification.
Consequently, PCA is not applied on the raw feature space.
We instead followed a feature filtering approach according to
[64] and used our FQ metric to identify the n best features in
the feature space. Also note that other feature reduction
approaches such as the supervised linear discriminant analysis
(LDA) can instead be used as a classifier on a feature space
based on manual feature selection.
We iterate through these features and check if a feature cor-
relation exceeds the critical threshold and either accept or
reject it for the reduced feature space. For the sake of compari-
son with the three features traction, roughness, fineness from
Fishel et al. [22], we choose to extract the three best features
to define X3 as well as the ten best features in a feature space
X10 . Table IV shows the feature names of the best features
from different modalities.
Table V shows the final correlations between the TacTUM
features defined in this article. For example, TCOTUM and
TPRTUM are strongly correlated, and hence, cannot both be
selected for the final feature space. Similar to adhesive tack
and yielding, both features presumably are highly relevant for
haptic display as they are related to the initial and persistent
thermal drain of energy, though.
B. Hand-Crafted Features Classifier Evaluation
Our model is trained and tested on the different sets of the
five-fold partitions. We further evaluate our classifiers based
on three different classification levels (coarse, medium, fine)
and two metrics (accuracy and F1 score). Coarse classification
refers to the discrimination of C, i.e., how the classification
engine is able to distinguish between the major classes C1 –
C8 regardless the sub-class S or the material M. Medium clas-
sification further includes the sub-class S, e.g., hardwood or
softwood, as shown in Table I. Fine classification additionally
considers the material M, such as aluminum, ABS, or oak. We
apply common classifiers such as support-vector machines
(SVM), linear discriminant analysis (LDA), k-nearest neighbor
(k-NN), neural network (NN), bagged trees, also known as ran-
dom forest (RF), and Naive Bayes (NB). The neural network is
trained using 60% training, 20% validation, and 20% test data
based on 500 hidden layers (tested with different numbers of
layers) and a scaled conjugate gradient back-propagation.
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
417
TABLE V
CRITICAL CORRELATIONS OF FINAL TACTUM FEATURES. NOTE THAT THE
FEATURES RELATED TO STIFFNESS ARE CORRELATED DUE TO THE FACT THAT
THE DATABASE CONTAINS MUCH MORE RIGID THAN DEFORMABLE MATERIALS,
AND HENCE, THE CORRESPONDING STIFFNESS FEATURE VALUES ARE SET TO 0
C. Results and Discussion
The accuracy of a classification engine is an intuitive metric
whenever classes are balanced [63]. We additionally calculate
the F1 scores (averaged per class) for each classifier, because
we face a class imbalance on a coarse and medium classifica-
tion level. Figs. 16 and 17 show the accuracies (top) and F1
scores (bottom) for the feature spaces with ten selected fea-
tures X10 from different domains, e.g., purely image-based
features.The F1 score is specifically more descriptive when-
ever faced with class imbalance, which is visible in Fig. 16. If
fine material classification based on all 184 material classes is
considered, this imbalance is not present anymore.
We further evaluate only the three best features of the tactile
domain as well as from our previous work in [28] and the fea-
tures (traction, roughness, fineness) presented by Fishel
et al. [22]. Note that these features are only mathematically
adapted from their original definition from [22] using our defini-
tions FRSTUM for traction, the logarithmic band-filtered signal
power for roughness (as in [22]), and MiCOTUM for fineness.
We observe that three features lead to an accuracy of about 30%
to 65% shown in Fig. 18. We can thereby confirm that the fea-
tures proposed by Fishel et al. [22] and the three best TacTUM
features show a noticeable classification performance. The rea-
son why the best three (best-3 from Table IV) features even out-
perform these results is caused by the best feature, the material
density. We conclude that whenever the density can be extracted
reliably, it significantly improves the classification performance
of any material scanning system and even outperforms human
classification performance. This does not come as a surprise. For
example, the densities of woods are remarkably different
whereas visual and tangible information can be easily confused
even by humans examining such material samples.
D. Deep Learning-Based Material Classification
Convolutional neural networks (CNNs) have been applied for
handwritten number classification by LeCun et al. [67]. Kriz-
hevsky et al. [68] further have proven the success of using
CNNs for image-based object recognition, and ever since,
several adaptations like VGGM by Cimpoi et al. [69] or
418
Fig. 16. Results of the coarse classification with X10 (top: accuracy, bot-
tom: F1 score) for different classifiers. The x-axis shows the sets of features
used to train the classifiers.
Fig. 18. Results of the fine classification with X3 (top: accuracy, bottom: F1
score) using only the three best features from different domains and related work.
VGGVD [70] have been proposed for surface texture classifica-
tion. Bello et al. [71] extensively compared deep networks with
hand-crafted image features. Deep networks outperform other
approaches in terms of texture classification whenever non-sta-
tionary textures and the presence of multiple acquisition condi-
tions changes is considered. By contrast, hand-crafted features
are superior in distinguishing stationary textures under steady
imaging conditions and have proven to be more robust than
CNN-based features for variation in the image rotation [71].
Due to the rapid progress of deep learning, a comparison to the
hand-crafted feature approach is inevitable in this context.
1) Network Architecture and Results: Liu et al. [57] sum-
marize that the network structures AlexNet, VGGM, VGGVD,
and TCNN literally solved the texture classification problem on
the largest existing texture data-sets such as CUReT [72], KTH-
TIPS2 [73], or ALOT [74] by achieving a classification perfor-
mance of 99% according to Cimpoi et al. [69]. That is why we
use our database images based in these previously defined net-
work structures and do not go further into the network design
details. Fig. 19 shows the DL network which we use in this arti-
cle. It is transfer-learned from AlexNet [68] which has proven to
work excellently for object recognition. We also cross-validated
the adapted network with the ALOT database (images sliced to a
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
Fig. 17. Results of the fine classification with X10 . The dotted line shows the
best accuracy of 90.2%
1.2% and F1 score of 0.90
0.01.
Fig. 19. DL network based on AlexNet [68] adapted to perform a fine classi-
fication of the 184 material classes in this article. Each convolutional layer is
followed by a batch normalization and rectified linear unit (ReLU) layer. If
not explicitly stated, the stride was set to 2 and the padding to “same”.
fix size of 768 x 330 x 3) and achieved a comparable result of
95%. The actual bottleneck is the amount of correctly labeled
data to learn from. Existing deep learning-based networks
almost perfectly classify the available texture image databases
as summarized in [57]. Consequently, we consider our extensive
image database with the new material labeling taxonomy as
highly suitable to run further validations. We sliced the captured
RGB images Idisp from Section III into [500 x 500] RGB patches
leading to 175 images per material class with no pixel overlap
between slices. We rotated each image by 90o to augment this
data-set, leading to 350 images per material class. We applied a
split of 80% training and 20% test data. Fig. 20 shows a few ran-
domly selected patches from our DL database which we denote
as TexTUM database.
We use the following hyper-parameters for training.
 Solver: stochastic gradient descent (SGDM) with
momentum of 0.9, alternatively: adam optimizer
 Mini batch size of 128
 Initial learning rate of 0.01
 L2 regularization of 0.0001,
 dropout layers with 40% dropout
Overall, the DL network achieves a classification accuracy
of 90.7%
1.0% after 40 epochs of training, and hence,
slightly outperforms all hand-crafted approaches.
2) Misclassification Comparison of Deep Learning and
Hand-Crafted Features: Overall, the image-based DL network
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
Fig. 20. Thirty exemplary [500 x 500] RGB material image patches for the
DL network. Overall, 64,400 of these texture patches are used to train (75%)
and test (25%) the DL network in this article to perform the fine classification
of the 184 materials constituting the largest taxonomy-based image texture
database to date (TexTUM database).
slightly outperforms the hand-crafted feature approach in terms
of classification accuracy. However, one critical point arises
considering content-based material classification. Several rele-
vant material characteristics, like the friction coefficient or the
compliance, cannot be learned by such an image-based deep
neural network, and hence, the system has a poor understanding
of haptic material properties. This observation can be proven by
considering the misclassified labels based on the true labels y
and the predictions y^. In order to account for this observation,
we count the overall misclassified labels N (with y^ ¼ 6 y) and
compute the ratio rC of the misclassified major class labels
y^C ¼
6 yC . We extract the class identifier y^C from each misclassi-
fication y^ and calculate
PN
n¼1 ð1  dy^C;n ;yC;n Þ
rC ¼ (32)
N
with dij being the Kronecker delta, which is equal to one if the
two integer arguments i and j (the predicted and true labels) are
the same. We consider it critical whenever the predicted class
label y^C is not equal to the true class label yC and then increase
the numerator by one. The denominator denotes the number of
all misclassifications N based on the fine classification material
labels. The best hand-crafted (HC) approach using the random
forest classifier leads to rC;HC ¼ 0.05, and the trained DL net-
work leads to rC;DL ¼ 0.24. Despite a better classification accu-
racy (i.e., less misclassifications), the labeling error of the DL
network is still larger than the one of the hand-crafted approach.
Intuitively, the image-based DL network faces a critical chal-
lenge while distinguishing between hard and soft materials and
has no notion of the characteristic microscopic roughness or
surface friction. By contrast, hand-crafted tactile features repre-
sent such features more robustly, but tend to confuse the materi-
als among each other. That is why we believe that both
approaches have their advantages and right of existence, and
ideally are fused into a hybrid approach in future work.
E. Discussion
We can draw the conclusion that a selection of features
from different modalities performs best when combined with
popular classifiers like Naive Bayes or random forest, which
also have been applied successfully to comparable classifica-
tion tasks. Ten features from different domains already lead to
a satisfying performance on different classification label detail
levels, and we observe even larger accuracies and F1 scores
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
419
using more content-based features. Until recently, extensive
feature design filled above 90% of the engineering work [65].
Deep learning-based approaches further confirm to be a valid
and promising alternative to the proposed hand-crafted feature
approaches [57], [68] in this article. Beyond purely image-
based approaches, deep neural networks have also been trained
on haptic-related data as shown by Gao et al. [75] and in our
previous work [76], [77], e.g., on spectrograms of vibro-tactile
signals to learn hidden features from the tactile domain. We
believe that a hybrid approach of prior knowledge (based on
feature design) and deep neural networks will ultimately com-
bine the advantages of both methodologies, which already
show high standalone classification performance. Up to this
point, even larger amounts of haptic-related data is required
which supports the fact that versatile object scanners (i.e.,
robot-based or operator-based) are crucial to capture the data
based on an appropriate material taxonomy. In this context we
would like to emphasize the relevance of taxonomies like the
one proposed; materials need to be unambiguously catego-
rized, notably according to their actual processing shape, to
ensure potential merging of haptic databases in the future.
VI. CONCLUSION
This article presented an improved material property acquisi-
tion device (Texplorer2), a novel material taxonomy and a new
multi-modal feature space definition. A complete pipeline was
proposed that can be extended with further materials due to the
flexible recording setup and the taxonomy. We defined a set of
tactile features based on human exploratory procedures, and
together with image and audio features, common classifiers
were trained to successfully identify the materials. The compar-
ison with image-based deep learning showed that they outper-
form hand-crafted approaches in terms of accuracy, but led to
obvious misclassifications in terms of haptic similarity. Ongo-
ing research in this field now needs to collect more data to rep-
resent all different processing shapes of the materials, and also
should prove if the fifteen tactile features are descriptive enough
for haptic display. We intend to use the fifteen-dimensional fea-
ture space as input parameters for haptic display. A separate
manuscript will present how the features generate the control
signals for a novel haptic display device. Subjective experi-
ments will be carried out to show the effectiveness in rendering
the tactile dimensions. Additionally, the Texplorer2 will be
equipped with a speed measurement unit, e.g., contact-less by
mounting permanent magnets and a magnetometer. These
measurements are required to constrain the operator motion
into a specific range as well as to provide vibro-tactile signals
of different scan speed for potential haptic display.
ACKNOWLEDGEMENT
We would like to thank the Chair of Building Construction and
Material Science (TUM Department of Architecture) for access
to the collection of building materials area, and specifically, Dipl.
Ing. Johann Weber for providing extensive information about the
materials. In addition, the authors would like to thank the anony-
mous reviewers for their constructive comments which helped to
improve this article.
420 IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020

APPENDIX

Fig. 21. Overview of all 184 material classes in the database.

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES 421

TABLE VI
MATERIAL CATEGORIZATION CONVENTION TABLE. FOR A COMPLETE LIST PLEASE REFER TO OUR WEBSITE [42]

Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
422
REFERENCES
[1] S. J. Lederman and R. L. Klatzky, “Haptic perception: A tutorial,” Atten-
tion, Perception, Psychophys., vol. 71, no. 7, pp. 1439–1459, 2009.
[2] S. Lacey, C. Campbell, and K. Sathian, “Vision and touch: Multiple or
multisensory representations of objects?,” Perception, vol. 36, no. 10,
pp. 1513–1521, 2007.
[3] Y. Vardar, C. Wallraven, and K. J. Kuchenbecker, “Fingertip interaction
metrics correlate with visual and haptic perception of real surfaces,” in
Proc. IEEE World Haptics Conf., 2019, pp. 395–400.
[4] K. Eilers, “D. Katz, Der Aufbau der Tastwelt (book review),”
Kant-Studien, vol. 31, p. 104 ff, 1926.
[5] S. J. Lederman, “Tactile roughness of grooved surfaces: The touching
process and effects of macro-and microsurface structure,” Perception
Psychophys., vol. 16, no. 2, pp. 385–395, 1974.
[6] R. L. Klatzky, S. J. Lederman, C. Hamilton, M. Grindley, and
R. H. Swendsen, “Feeling textures through a probe: Effects of probe
and surface geometry and exploratory factors,” Perception Psychophys.,
vol. 65, no. 4, pp. 613–631, 2003.
[7] S. J. Bensma€ıa and M. Hollins, “The vibrations of texture,” Somatosen-
sory Motor Res., vol. 20, no. 1, pp. 33–43, 2003.
[8] S. J. Bensma€ıa and M. Hollins, “Pacinian representations of fine
surface texture,” Perception Psychophys., vol. 67, no. 5, pp. 842–
854, 2005.
[9] W. M. Bergmann Tiest and A. M. Kappers, “Haptic and visual
perception of roughness,” Acta Psychol., vol. 124, no. 2, pp. 177–
189, 2007.
[10] W. M. Bergmann Tiest, “Tactual perception of material properties,”
Vision Res., vol. 50, no. 24, pp. 2775–82, 2010.
[11] C. E. Schroeder, J. Smiley, K. G. Fu, T. McGinnis, M. N. O’Connell,
and T. A. Hackett, “Anatomical mechanisms and functional implications
of multisensory convergence in early cortical processing,” Int. J. Psy-
chophysiol., vol. 50, no. 1–2, pp. 5–17, 2003.
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
[12] J. M. Yau, J. B. Olenczak, J. F. Dammann, and S. J. Bensma€ıa,
“Temporal frequency channels are linked across audition and touch,”
Current Biol., vol. 19, no. 7, pp. 561–566, 2009.
[13] J. A. Fishel and J. E. Colgate. Haptics symposium 2018 cross-cutting
challenge: Haptic dimensions of surfaces. (2019). [Online]. Available:
http://2018.hapticssymposium.org/ccc2
[14] S. Okamoto, H. Nagano, and Y. Yamada, “Psychophysical dimensions
of tactile perception of textures,” IEEE Trans. Haptics, vol. 6, no. 1,
pp. 81–93, First Quarter 2013.
[15] SynTouch. (2019). Syntouch Inc. website. [Online]. Available: https://
www.syntouchinc.com/en/
[16] J. A. Fishel, G. E. Loeb, B. Matulevich, and R. Davoodi, “Method and
applications for measurement of object tactile properties based on how
they likely feel to humans,” U.S. Patent 1 479 6647, 2016.
[17] K. J. Kuchenbecker, J. M. Romano, and W. McMahan, “Haptography:
Capturing and recreating the rich feel of real surfaces,” in Robot.
Research. Berlin, Germany: Springer, 2011, pp. 245–260.
[18] J. M. Romano, T. Yoshioka, and K. J. Kuchenbecker, “Automatic filter
design for synthesis of haptic textures from recorded acceleration data,”
in Proc. IEEE Int. Conf. Robot. Autom., 2010, pp. 1815–1821.
[19] H. Culbertson, J. Unwin, and K. J. Kuchenbecker, “Modeling and
rendering realistic textures from unconstrained tool-surface inter-
actions,” IEEE Trans. Haptics, vol. 7, no. 3, pp. 381–393, Jul.–Sep.
2014.
[20] H. Culbertson and K. J. Kuchenbecker, “Importance of matching
physical friction, hardness, and texture in creating realistic haptic virtual
surfaces,” IEEE Trans. Haptics, vol. 10, no. 1, pp. 63–74, Jan.–Mar.
2017.
[21] N. Jamali and C. Sammut, “Majority voting: Material classification by
tactile sensing using surface texture,” IEEE Trans. Robot., vol. 27,
no. 3, pp. 508–521, Jun. 2011.
[22] J. A. Fishel and G. E. Loeb, “Bayesian exploration for intelligent identi-
fication of textures,” Frontiers Neurorobot., vol. 6, p. 4, 2012.
STRESE et al.: HAPTIC MATERIAL ANALYSIS AND CLASSIFICATION INSPIRED BY HUMAN EXPLORATORY PROCEDURES
[23] J. M. Romano and K. J. Kuchenbecker, “Methods for robotic tool-
mediated haptic surface recognition,” in Proc. IEEE Haptics Symp.,
2014, pp. 49–56.
[24] J. Hoelscher, J. Peters, and T. Hermans, “Evaluation of tactile feature
extraction for interactive object recognition.,” in Proc. Humanoids,
2015, pp. 310–317.
[25] M. Kaboli and G. Cheng, “Robust tactile descriptors for discriminating
objects from textural properties via artificial robotic skin,” IEEE Trans.
Robot., vol. 34, no. 4, pp. 985–1003, Aug. 2018.
[26] A. Burka et al., “Proton: A visuo-haptic data acquisition system for
robotic learning of surface properties,” in Proc. IEEE Int. Conf. Multi-
sensor Fusion Integrat. Intell. Syst., 2016, pp. 58–65.
[27] M. Strese, C. Schuwerk, A. Iepure, and E. Steinbach, “Multimodal
feature-based surface material classification,” IEEE Trans. Haptics,
vol. 10, no. 2, pp. 226–239, Apr.–Jun. 2017.
[28] M. Strese, Y. Boeck, and E. Steinbach, “Content-based surface
material retrieval,” in Proc. IEEE World Haptics Conf., 2017,
pp. 352–357.
[29] S. Sato, S. Okamoto, Y. Matsuura, and Y. Yamada, “Wearable finger
pad sensor for tactile textures using propagated deformation on a side of
a finger: Assessment of accuracy,” in Proc. IEEE Int. Conf. Syst., Man,
Cybern., 2015, pp. 892–896.
[30] Y. Vardar, B. G€ uçl€
u, and C. Basdogan, “Effect of waveform on tactile
perception by electrovibration displayed on touch screens,” IEEE Trans.
Haptics, vol. 10, no. 4, pp. 488–499, Oct.–Dec. 2017.
[31] Y. Visell and Y. Shao, “Learning constituent parts of touch stimuli
from whole hand vibrations,” in Proc. IEEE Haptics Symp., 2016,
pp. 253–258.
[32] R. Grigorii, M. Peshkin, and J. E. Colgate, “High-bandwidth tribometry
as a means of recording natural textures,” in Proc. IEEE World Haptics
Conf., Munich, Germany, 2017, pp. 629–634.
[33] S. Shin and S. Choi, “Geometry-based haptic texture modeling and
rendering using photometric stereo,” in Proc. IEEE Haptics Symp.,
San Francisco, CA, USA, 2018, pp. 262–269.
[34] M. Dulik and L. Ladanyi, “Surface detection and recognition using
infrared light,” in Proc. IEEE Elektro, 2014, pp. 159–164.
[35] T. Aujeszky, G. Korres, and M. Eid, “Measurement-based thermal
modeling using laser thermography,” IEEE Trans. Instrum. Meas.,
vol. 67, no. 6, pp. 1359–1369, Jun. 2018.
[36] H. Choi, S. Cho, S. Shin, H. Lee, and S. Choi, “Data-driven thermal ren-
dering: An initial study,” in Proc. IEEE Haptics Symp., San Francisco,
CA, USA, 2018, pp. 344–350.
[37] C. Basdogan and M. A. Srinivasan, “Haptic rendering in virtual
environments,” Handbook Virtual Environ., vol. 1, pp. 117–134,
2002.
[38] A. M. Okamura, J. T. Dennerlein, and R. D. Howe, “Vibration feedback
models for virtual environments,” in Proc. IEEE Int. Conf. Robot. Auto-
mat., 1998, vol. 1, pp. 674–679.
[39] K. J. Kuchenbecker, J. Fiene, and G. Niemeyer, “Improving contact real-
ism through event-based haptic feedback,” IEEE Trans. Visual. Comput.
Graph., vol. 12, no. 2, pp. 219–230, Mar./Apr. 2006.
[40] J. Sinapov and V. Sukhoy, “Vibrotactile recognition and categorization
of surfaces by a humanoid robot,” IEEE Trans. Robot., vol. 27, no. 3,
pp. 488–497, Jun. 2011.
[41] V. Chu et al., “Robotic learning of haptic adjectives through physical
interaction,” Robot. Auton. Syst., vol. 63, pp. 279–292, 2015.
[42] M. Strese, C. Schuwerk, R. Chaudhari, and E. Steinbach, “Haptic texture
database,” [Online]. Available: https://zeus.lmt.ei.tum.de/downloads/
texture/
[43] H. Culbertson, J. J. Lopez Delgado, and K. J. Kuchenbecker, “One
hundred data-driven haptic texture models and open-source methods
for rendering on 3d objects,” in Proc. IEEE Haptics Symp., 2014,
pp. 319–325.
[44] W. Hassan, A. Abdulali, M. Abdullah, S. C. Ahn, and S. Jeon, “Towards
universal haptic library: Library-based haptic texture assignment using
image texture and perceptual space,” IEEE Trans. Haptics, vol. 11,
no. 2, pp. 291–303, Apr.–Jun. 2018.
[45] F. Musso and J. Weber, TUM collection of building materials. (2019).
[Online]. Available: https://www.ar.tum.de/en/ebb/collection-of-build-
ing-materials/
[46] H.-J. Bargel and G. Schulze, Werkstoffkunde. Berlin, Germany:
Springer-Verlag, 2008.
[47] N. Landin, J. M. Romano, W. McMahan, and K. J. Kuchenbecker,
“Dimensional reduction of high-frequency accelerations for haptic ren-
dering,” in Proc. Int. Conf. Human Haptic Sensing Touch Enabled
Comput. Appl., Springer, 2010, pp. 79–86.
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
423
[48] H. Culbertson, J. M. Romano, P. Castillo, M. Mintz, and
K. J. Kuchenbecker, “Refined methods for creating realistic haptic vir-
tual textures from tool-mediated contact acceleration data,” in Proc.
IEEE Haptics Symp., 2012, pp. 385–391.
[49] E. Meier. The wood database. (2019). [Online]. Available: https://www.
wood-database.com/
[50] Material Archiv. (2019). [Online]. Available: http://www.materia-
larchiv.ch/
[51] A. Burka and K. J. Kuchenbecker, “Handling scan-time parameters in
haptic surface classification,” in Proc. IEEE World Haptics Conf., 2017,
pp. 424–429.
[52] C. Harris and M. Stephens, “A combined corner and edge
detector.” in Proc. Alvey Vision Conf., Citeseer, 1988, vol. 15, no. 50.
pp. 1–5.
[53] H. Bay, T. Tuytelaars, and L. Van Gool, “Speeded up robust features
(SURF),” in Proc. Eur. Conf. Comput. Vision., Springer, 2006, pp. 404–
417.
[54] H. Tamura, S. Mori, and T. Yamawaki, “Textural features correspond-
ing to visual perception,” IEEE Trans. Syst., Man, Cybern., vol. 8, no. 6,
pp. 460–473, Jun. 1978.
[55] J. P. Den Hartog, Mechanical Vibrations. Courier Corporation, p. 55 ff,
1985.
[56] W. M. Bergmann Tiest and A. M. Kappers, “Tactile perception of
thermal diffusivity,” Attention Perception Psychophys., vol. 71, no. 3,
pp. 481–489, 2009.
[57] L. Liu, J. Chen, P. Fieguth, G. Zhao, R. Chellappa, and M. Pietik€ainen,
“From BoW to CNN: Two decades of texture representation for tex-
ture classification,” Int. J. Comput. Vision, vol. 127, no. 1, pp. 74–109,
2019.
[58] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural features
for image classification,” IEEE Trans. Systems, Man, Cybern., vol.
SMC-3, no. 6, pp. 610–621, Nov. 1973.
[59] R. Monzel. Haralick texture features Matlab. (2019). [Online]. Avail-
able: https://de.mathworks.com/matlabcentral/fileexchange/58769-hara-
licktextu refeatures
[60] S. Sornapudi. Tamura features. (2019). [Online]. Available: https://
github.com/Sdhir/TamuraFeatures
[61] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale
and rotation invariant texture classification with local binary patterns,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971–987,
Jul. 2002.
[62] T. Giannakopoulos and A. Pikrakis. Introduction to audio analysis.
(2019). [Online]. Available: https://de.mathworks.com/matlabcentral/
fileexchange/45831-matlab-audio- analysis-library
[63] P. Flach, Machine Learning: The Art and Science of Algorithms That
Make Sense of Data. Cambridge, U.K.: Cambridge University Press,
2012.
[64] I. Guyon and A. Elisseeff, “An introduction to variable and feature
selection,” J. Mach. Learn. Res., vol. 3, no. Mar., pp. 1157–1182, 2003.
[65] P. Domingos, “A few useful things to know about machine learning,”
Commun. ACM, vol. 55, no. 10, pp. 78–87, 2012.
[66] M. Strese, C. Schuwerk, and E. Steinbach, “Surface classification
using acceleration signals recorded during human freehand move-
ment,” in Proc. IEEE World Haptics Conf., Evanston, IL, USA, 2015,
pp. 214–219.
[67] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998.
[68] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., 2012, pp. 1097–1105.
[69] M. Cimpoi, S. Maji, and A. Vedaldi, “Deep filter banks for texture
recognition and segmentation,” in Proc. IEEE Conf. Comput. Vision
Pattern Recognit., 2015, pp. 3828–3836.
[70] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” 2014, arXiv:1409.1556.
[71] R. Bello-Cerezo, F. Bianconi, F. Di Maria, P. Napoletano, and
F. Smeraldi, “Comparative evaluation of hand-crafted image descriptors
vs. off-the-shelf CNN-based features for colour texture classification
under ideal and realistic conditions,” Appl. Sci., vol. 9, no. 4, p. 738, 2019.
[72] K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink,
“Reflectance and texture of real-world surfaces,” ACM Trans. Graph.,
vol. 18, no. 1, pp. 1–34, 1999.
[73] P. Mallikarjuna, A. T. Targhi, M. Fritz, E. Hayman, B. Caputo, and
J.-O. Eklundh, The KTH-TIPS2 Database. Stockholm, Sweden: Compu-
tational Vision and Active Perception Laboratory (CVAP), 2006.
424
[74] J. M. Geusebbroek, Amsterdam library of textures (ALOT). (2019).
[Online]. Available: http://aloi.science.uva.nl/public/alot/
[75] Y. Gao, L. A. Hendricks, K. J. Kuchenbecker, and T. Darrell, “Deep
learning for tactile understanding from visual and haptic data,” in Proc.
IEEE Int. Conf. Robot. Automat., 2016, pp. 536–543.
[76] M. Ji, L. Fang, H. Zheng, M. Strese, and E. Steinbach, “Preprocessing-
free surface material classification using convolutional neural networks
pretrained by sparse autoencoder,” in Proc. IEEE 25th Int. Workshop
Mach. Learn. Signal Process., 2015, pp. 1–6.
€
[77] H. Zheng, L. Fang, M. Ji, M. Strese, Y. Ozer, and E. Steinbach, “Deep
learning for surface material classification using haptic and visual
information,” IEEE Trans. Multimedia, vol. 18, no. 12, pp. 2407–2416,
Dec. 2016.
Matti Strese received the degree in electrical engi-
neering from the Technical University of Munich,
Germany. He received the Master of Science degree
in July 2014. After graduating, he joined the Chair of
Media Technology, Technical University of Munich,
in September 2014, where he is a member of the
Research Staff and working toward the Ph.D. degree.
His current research interests include the analysis of
haptic texture signals, surface classification, and arti-
ficial surface synthesis devices.
Lara Brudermueller received the Bachelor of Sci-
ence degree in management and technology with spe-
cialization in electrical engineering from the
Technical University of Munich, Germany. She is
currently working toward Master of Science degree
in robotics, cognition, and intelligence from the
Technical University of Munich. In the summer of
2018, she wrote her bachelor’s thesis in the field of
haptics and object surface recognition at the Chair of
Media Technology.
Authorized licensed use limited to: SLUB Dresden. Downloaded on June 02,2023 at 07:51:49 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON HAPTICS, VOL. 13, NO. 2, APRIL-JUNE 2020
Jonas Kirsch received the Bachelor of Science
degree in electrical and computer engineering from
the Technical University of Munich and is currently
working toward the Master of Science degree. In the
summer of 2018, he completed his bachelor thesis in
the field of haptics and tactile data acquisition and
has been working ever since at the Chair of Media
Technology.
Eckehard Steinbach (M’96–SM’08–F’15) received
the degree in electrical engineering from the Univer-
sity of Karlsruhe (Germany), University of Essex,
Britain, and ESIEE in Paris. From 1994 to 2000,
he was a member of the Research Staff of the Image
Communication Group, University of Erlangen-
Nuremberg, Germany, where he received the engi-
neering doctorate in 1999. From February 2000 to
December 2001, he was a Postdoctoral Fellow with
the Information Systems Laboratory of Stanford Uni-
versity. In February 2002, he joined the Department
of Electrical and Computer Engineering, Technical University of Munich,
Germany, where he is currently a Full Professor of Media Technology. His
current research interests include the area of haptic and visual communication,
indoor mapping, and localization.