This article has been accepted for publication in IEEE Transactions on Multimedia.
This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
Perception-Aware Cross-Modal Signal
Reconstruction: From Audio-Haptic to Visual
Xin Wei, Member, IEEE, Yuyuan Yao, Haoyu Wang, and Liang Zhou, Senior Member, IEEE
Abstract—Cross-modal communications, devoting to collabora-
tively delivering and processing audio, visual, and haptic signals,
have gradually become the supporting technology for the emerg-
ing multi-modal services. However, the inevitable resource compe-
titions among different modality signals as well as the unexpected
packet loss and latency during transmission seriously affect qual-
ity of the received signals and end user’s immersive experience
(especially visual experience). To overcome these dilemmas, this
paper proposes a cross-modal signal reconstruction strategy from
the perspective of human’s perceptual facts. It tries to guarantee
visual signal quality by considering potential correlations among
modalities when processing audio and haptic signals. On the one
hand, a time-frequency masking-based audio-haptic redundancy
elimination mechanism is designed by resorting to the similarity
of audio-haptic characteristics and human’s masking effects. On
the other hand, based on the fact that non-visual perception can
assist to form and enhance visual perception, an audio-haptic
fused visual signal restoration (AHFVR) approach for handling
the impaired and delayed visual signals is proposed. Experiments
on a standard multi-modal database and a constructed practical
platform evaluate the performance of the proposed perception-
aware cross-modal signal reconstruction strategy.
Index Terms—cross-modal communications, perception, visual
signal reconstruction, audio-haptic redundancy elimination.
I. I NTRODUCTION
W ITH the dramatic development of wireless communi-
cations and multimedia technologies, humans begin to
pursue more extreme interaction and immersion after their
audio-visual sensation has been satisfied [1]. Accordingly,
multi-modal services, typically integrating audio, visual, and
haptic signals, will obviously provide a much more immersive
experience in various scenarios, such as entertainment, educa-
tion, healthcare, and industry [2]. For example, haptic-enabled
telesurgery will improve the doctor’s interaction and scene
experiences when providing 360 degree panoramic video, high
definition audio, and force feedback as well as guaranteeing
transmission and signal processing issues [3].
In order to support multi-modal services and enhance end
user’s immersive experience, cross-modal communications
come into being [4], [5]. Compared with existing multi-
modal communication schemes [1], [6], it tries to leverage the
Manuscript received April 4, 2022; revised June 14, 2022; accepted July
12, 2022. This work was supported by the National Natural Science Foun-
dation of China under Grant 62071254 and the Priority Academic Program
Development of Jiangsu Higher Education Institutions. The associate editor
coordinating the review of this manuscript and approving it for publication
was Dr. Zhi Wang. (Corresponding Author: Liang Zhou.)
The authors are with School of Communications and Information Engi-
neering, and Key Lab of Broadband Wireless Communication and Sensor
Network Technology, Ministry of Education, Nanjing University of Posts and
Telecommunications, Nanjing 210003, China. (E-mails: {xwei, 1019010631,
1221014002, liang.zhou}@njupt.edu.cn).
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
1
potential correlation among modalities to satisfy demands such
as low-latency, high reliability, high throughput. However,
when transmitting and receiving audio, visual, and haptic
information under the existing cross-modal communication
framework, it still faces two main challenges. First, haptic,
audio, and visual streams inevitably compete for the limited
resources during transmission, which especially has much
bad effect on visual streams. Existing modality priority-based
scheduling schemes give haptic stream higher priority to
guarantee its low latency and high reliability requirements (see
Table I) [3], [6]. In other words, when haptic stream appears,
visual streams with lower priority should be interrupted. After
the transmission of the high-priority haptic stream has finished,
visual stream is resumed. Though haptic stream has a smaller
scale compared to visual stream, its high occurrence frequency
and bursty characteristic still severely affect the quality of
visual streaming [7].
TABLE I: Transmission requirements of haptic, visual, and
audio signals [4]
Haptic Visual Audio
Delay [1 − 60]ms ≤ 350ms ≤ 150ms
Jitter [1 − 10]ms ≤ 30ms ≤ 30ms
∼ 10−5 %
Data loss rate
Data rate ∼ 128Kbps
∼ 20%
[2.5 − 40]M bps
∼ 30%
[22 − 200]Kbps
Second, due to unexpected packet loss and transmission
latency, the received quality of visual signals can hardly
be satisfied. During multi-modal streaming, the changeable
transmission environment unavoidably provides side effects,
especially for the received visual signals [8]. The reason is
that, visual streams are obviously large scale which may
cause impairment during receiving. Moreover, owing to the
lower transmission priority and the limited bandwidth, the
delayed arrival of visual streams is prone to happen. Tradi-
tional methods such as restoration within visual modality or
retransmission of the destroyed streams are not suitable for
solving this issue, which may influence the synchronization
between visual and non-visual (haptic, audio) modalities and
user’s immersive experience [9], [10].
To deal with the above issues, cross-modal signal recon-
struction strategy is considered as an effective way [4]. Before
transmission, the volume of non-visual signals should be
reduced as much as possible to guarantee visual transmission
throughput. For example, if the redundancy of haptic signals
is analyzed and the occurrence frequency can be reasonably
reduced, the competition between haptic and visual streams
can be weaken and the throughput of visual streaming will
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
be improved. After receiving, if visual impairment or delayed
phenomena emerge, restoration approach can be adopted by
taking advantage of potential correlations between visual and
non-visual signals. It is considered as an effective substitution
for retransmission scheme.
In the human’s perception system, there exist inherent inter-
actions among auditory, haptic, and visual sensory subsystems,
which comprehensively determines human’s experience [11],
[12]. As the ultimate goal of multi-modal services is to
promote user’s immersive sensory experience, it motivates us
to borrow the related mechanisms from the human’s percep-
tion system when designing cross-modal signal reconstruction
architecture. Specifically, as two representative non-visual
perceptions, there are plenty of similarities between audio and
haptic sensories. On the one hand, audio and haptic signals
not only have relatively small size requiring low bandwidth
(∼ Kbps in Table I), but also share the similar waveforms
in time and frequency domains for the specific content (e.
g. wood texture in Fig. 1). On the other hand, humans have
analogous masking effects both in their auditory and haptic
subsystems [13], [14]. The former fact inspires us whether
to take audio and haptic signals together, while the latter
phenomenon stimulates us to reduce the volume of these two
non-visual signals. In other words, considering and eliminating
redundancy of these two non-visual signals together may
be suitable for enhancing visual streaming efficiency at the
sender, which has not been attracted attention in current works.
Fig. 1: Use case of the audio and haptic signals of wood
texture in time and frequency domains.
Moreover, in psychological theory, it shows that humans’
non-visual perception can assist in forming and enhancing
their visual experiences [15]. For instance, blind persons or
infants can recognize and reconstruct visual forms of envi-
ronments by hearing and touching the related objects after
conscious training. The reason is that there exists intrinsic
content consistency among multi-modal sensory signals [16],
[17]. This idea can be transferred to restore the impaired
and delayed visual signals at the receiver, where multi-modal
signals co-exist and also have potential correlations. However,
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
2
how to explore and leverage these correlations among modal-
ities to realize visual signal restoration is still difficult.
Based on the above analysis, we aim at designing a cross-
modal reconstruction strategy by resorting to the mechanisms
in human’s perception. The contributions of this paper are
listed as follows:
1) We construct a perception-aware cross-modal signal
reconstruction architecture. It elaborately contains an audio-
haptic redundancy elimination mechanism at the sender as well
as an audio-haptic fused visual signal restoration approach at
the receiver.
2) We design a time-frequency masking-based redundancy
elimination mechanism for audio and haptic streams to be
delivered at the sender. It takes advantage of masking effect to
reduce the occurrence frequency of audio and haptic streams.
3) We propose an audio-haptic fused visual signal restora-
tion (AHFVR) approach. It utilizes modality correlation for
handling the impaired or delayed visual signals at the re-
ceiver. In this approach, low-level features extracted from the
received audio and haptic signals are first fused with semantic
constraints. Then, an adversarial scheme is introduced for
the fused features to capture its latent correlation in real
visual space. Finally, a hierarchical fine-grained representation
structure and the knowledge distillation technique are adopted
to realize the desired visual signal generation.
The rest of this paper is arranged as follows: Section II
introduces the related work. Section III describes the construct-
ed perception-aware cross-modal signal reconstruction archi-
tecture. The details of the designed time-frequency masking-
based audio-haptic redundancy elimination mechanism and the
proposed AHFVR approach are presented in Section IV and
Section V, respectively. Experimental results and analysis in a
standard dataset are given in Section VI. In Section VII, the
effectiveness of the proposed perception-aware cross-modal
signal reconstruction is verified in a practical platform. Finally,
Section VIII gives concluding remarks.
II. R ELATED W ORK
A. Multi-Modal Stream Transmission
When considering multi-modal stream transmission, modal-
ity priority assignment technique is widely studied for allocat-
ing available transmission rates [18]. Although this scheme
can well handle the delivery of multi-modal streams, it is
not adaptive to dynamic changes of networks. Subsequently,
multiplexing is another probably suitable solution for resource
allocation. Typically, the work in [19] investigates an adaptive
application layer multiplexing framework concerning multi-
modal streams. In [6], a multiplexing scheme for multi-modal
feedback (visual, audio and haptic) is proposed to give high
priority to the haptic data. Moreover, a preemptive-resume
scheduling strategy is also adopted for multi-modal stream-
ing. To further consider characteristics among modalities, we
propose the architecture of cross-modal communications and
further design the cross-modal scheduling strategy to collab-
oratively allocate resources for the specific stream according
to application requirements [4], [3].
In summary, the main aim of the current transmission strate-
gies is to guarantee haptic stream delivery with a relatively
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
Master Domain
terminal edge node
haptic Intact
decoder
haptic
impaired
touch
delayed
audio
headphone
visual
display Audio-Haptic Fused
screen Visual Signal Restoration
Fig. 2: The perception-aware cross-modal signal reconstruction architecture
high priority. The other modality streams (audio and visual
streams) are taken as a whole when designing delivery proto-
col. Moreover, different compression schemes are respectively
used for three modality streams, affecting the efficiency of the
delivery scheme. Recently, perceptual redundancy elimination
techniques which investigate masking threshold or just no-
ticeable distortion have been taken for either audio or haptic
signals [20], [21]. However, they are rarely considered as a
joint problem, which is one of the main purposes in this paper.
B. Visual Signal Restoration
In [22], the previous research and application of visu-
al perception in different industrial fields such as product
surface defect detection, intelligent agricultural production,
intelligent driving, image synthesis and event reconstruction
are reviewed. Visual signal restoration is important and d-
ifficult in multi-modal signal processing, especially visual
perception. To effectively eliminate visual distortions, a semi-
supervised learning-based model and optimization approach
is designed [23]. In [24], a sparse optimization approach
named `0 TV-PADMM is proposed, which takes advantage of
total variation to deal with noise errors. In [25], it relates
the noise to the semantic segments and presents an efficient
approach to diverse image synthesis. Besides noise, packet
loss occurring in the wireless network can also jeopardize
the performance of visual streams. In [26], a multi-channel
error recovery approach is proposed to guarantee high-quality
and real-time video streaming. It integrates priority queue,
quick start, and scalable reliable channel to overcome the high
packet loss ratio. In [27], a deep learning-based restoration
algorithm is developed for handling packet loss in wireless
multimedia sensor network, which reconstructs the impaired
visual frames with information from neighboring available
frames. In addition to noise and packet loss, interference from
transmission environments should not be neglected. To restore
images with interference, [28] utilizes common dark channel
prior and incorporates adaptive color correction for removing
color casts.
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
3
Network Domain Slave Domain
wireless network edge node terminal
haptic
visual reduction label tagging
robot arm
priority audio
multiplexing assignment
microphone
visual
Time-Frequency Masking-
based Audio-Haptic
Redundancy Elimination camera
Based on the above analysis, when considering visual signal
restoration, the majority of current approaches rely on the
visual modality itself [29]. Due to the severe content loss or
delay of desired visual streams during transmission, the results
will be far from satisfactory. To handle this issue, cross-modal-
based approaches have been considered. In [30], a cross-
modality transfer learning algorithm is designed to connect
image and text data, enhancing accuracy of classification task
with the aid of modality transfer. In [8], it tries to restore
visual signals by using haptic modality. However, it does not
consider the co-existence of audio signals. It also does not
design schemes from the perspective of human’s perceptual
facts, which are valuable and meaningful. As a result, how to
realize effective visual signal restoration from the other non-
visual modalities is still the focus in this paper.
III. P ERCEPTION -AWARE C ROSS -M ODAL S IGNAL
R ECONSTRUCTION A RCHITECTURE
As visualized in Fig. 2, the constructed perception-aware
cross-modal signal reconstruction architecture can be decom-
posed into three parts: master domain, network domain, and
slave domain. The master domain at the receiver represents
a human operator and a human system interface (HSI). The
HSI consists of an input device such as Geomagic Touch
for haptic positioning and orientating, and output devices for
multi-modal displaying, e.g., haptic device for force feedback,
video display, and headphones. The slave domain at the sender
contains a controlled robot arm equipped with multiple haptic
sensors, a video camera and a microphone. The network
domain connecting them provides the communication medium.
Controlled by the master domain, the slave domain is able
to directly sense and interact with the remote environment
and return multi-modal feedback over the network domain.
Moreover, motivated by the benefits of edge intelligence [5],
we deploy edge nodes in both master and slave domains to
deal with multi-modal signal processing.
When the devices at the slave terminal collect audito-
ry, visual, and touch stimuli from surroundings or get the
commands from the master terminal, the related multi-modal
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 4

signals are initially uploaded to the edge node for modality

variance-
and category labeling, forming streams. The labeled streams signal
based
energy
are then divided into two categories: audio-haptic streams and feature
estimation
audio blocking extraction quantization audio
visual streams, which is different from existing schemes. The signal stream
reason is that audio and haptic signals have similar character- discrete time-frequency
istics and transmission requirements. Then, a joint redundancy haptic wavelet masking threshold entropy haptic
signal transform seeking coding stream
elimination mechanism (described in Section IV) is designed
preprocessing compression
for the audio-haptic streams from the perspective of human’s masking ratio
masking effect, which can enhance the efficiency of visual determination
streams to be delivered. Finally, the priority assignment and Fig. 4: Procedure of the proposed audio-haptic redundancy
multiplexing methods are adopted, which gives the compressed elimination mechanism
audio-haptic streams higher priority compared with visual
streams.
When the multi-modal packets have been delivered via
wireless channels and received by the edge node managing the
master terminal, the corresponding signals are first decoded
from packets. Due to the relatively low priority and trans-
mission guarantee, if visual signals are impaired or delayed,
signal restoration (described in Section V) can be adopted by
exploring and leveraging intrinsic content consistency among
modalities. Finally, the intact or restored visual signals as well
as the received audio and haptic signals are transferred from
the edge node to the associated devices of master terminal.

IV. T IME -F REQUENCY M ASKING -BASED AUDIO -H APTIC

R EDUNDANCY E LIMINATION M ECHANISM (a)
6 0
The main idea of masking effect comes from theory about th e p ro p o se d sc h e m e (a u d io )
human perception. It is to remove the perceptual redundancies 5 5
th e p ro p o se d sc h e m e (h a p tic )
while causing no difference to the original signals. More sc h e m e in [ 6 ] (a u d io )
sc h e m e in [ 6 ] (h a p tic )
precisely, it implies that components of audio signals or haptic 5 0
signals with a strong frequency will mask components with a
weak frequency. As a result, the weak frequency components
P S N R

4 5
having no sense for transmission can be filtered. Moreover,
frequency masking is extensively investigated in practical 4 0
applications while time masking phenomenon is rarely imple-
mented quantitatively. Actually, time masking, especially the 3 5
backward masking, cannot be ignored due to its long duration
and large masking amount, which is essential for compression 3 0
5 1 0 1 5 2 0 2 5 3 0 3 5
[31]. Therefore, it is necessary to joint consideration between c o m p re s s io n ra tio
frequency masking and time masking.
(b)

Fig. 5: (a) Average throughput; b) PSNR of the proposed

redundancy elimination mechanism and that in [6]
5

4 masking unit
masking threshold Inspired by this, to enhance transmission efficiency and
3
amplitude

guarantee receiving efficiency, we consider audio and haptic

2 streams together and design a joint time-frequency masking-
1 the masked
based redundancy elimination strategy at the sender. Specif-
units ically, the whole procedure is conducted in time-frequency
0
10 domain, where time and frequency information concerning
10 global and local attributes can be simultaneously captured,
5 8
6
4 shown in Fig. 3. In this figure, the two-dimensional curved
2
frequency(Hz) 0 0 time(s)
surface represents masking threshold, and the green solid
circles are the masking signals. The blue solid circles whose
Fig. 3: Illustration of masking effect in time-frequency amplitude less than the masking threshold can be masked,
domains while the red ones are selected for transmission because their

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 5

received real visual

audio signals A signals V

Fa a v
Fv
D1 0/1 hierarchical D2 0/1
constraint

received f
H Ff
Vˆ
generated
haptic signals
visual signals
GRU

Fh h G
audio-haptic modality irrelevant cross-modal visual
semantic fusion latent space learning signal generation

Fig. 6: Illustration of the proposed audio-haptic fused visual signal restoration (AHFVR) approach

amplitudes are higher than the threshold. Furthermore, Fig.
4 completely depicts the diagram of the proposed scheme.
Concretely, the audio and haptic signals are first switched
into time-frequency representations through wavelet transform.
This can provide more features compared with sole time
or frequency domains. The obtained time-frequency plane is
filtered, denoised, and further utilized to extract variance-based
features, while the signal energy is measured at every time-
frequency unit at the same time. Then, an unsupervised and
nonparametric adaptive method is adopted to estimate the
masking threshold via time-frequency vector [32], associating
to the previous and future frames of the current signals. It is
noted that this masking threshold is dynamic according to the
actual characteristics of audio and haptic signals. By obtaining
masking threshold, the signal masking ratio or compression
ratio can be determined. Finally, this ratio is applied to every
time-frequency unit for quantization and entropy coding. If
the audio and haptic features in the time-frequency space
are lower their respective masking threshold, they should not
be transmitted. As a result, the redundancy of audio and
haptic signals will be greatly eliminated or compressed without
feeling the perceptual difference.
We apply the above proposed redundancy elimination mech-
anism under the constructed architecture in Section III. The
results are shown in Fig. 5. First, the average throughput of
multi-modal streams during a certain period of time is given
in Fig. 5(a). On the one hand, the transmission efficiency
of audio and haptic modality streams can be promoted due
to redundancy elimination of these two modalities. On the
other hand, as these two modality streams have been com-
pressed, the resource competition between audio-haptic and
visual modalities can be weakened. It leads to the throughput
improvement of visual stream when compared with the scheme
in [6]. Moreover, in Fig. 5(b), it clearly shows that the
perceived quality of the compressed audio and haptic signals
degrades gracefully as higher compression ratios kick in.
However, the proposed redundancy elimination mechanism has
better performance than that in [6]. The reason is that the
proposed scheme uses time-frequency masking effects for both
audio and haptic modalities, achieving joint optimization under
cross-modal communication scenarios.
V. AHFVR:AUDIO -H APTIC F USED V ISUAL S IGNAL
R ESTORATION A PPROACH
In this section, to remedy the impaired or delayed visual
signals, we propose a simple yet effective approach named
audio-haptic fused visual signal restoration (AHFVR) ap-
proach. Its core idea comes from the psychological theory
that humans’ non-visual perception can assist in forming and
enhancing their visual experience. As illustrated in Fig. 6, the
overall approach can be further divided into three components:
audio-haptic semantic fusion, modality irrelevant latent space
learning, cross-modal visual signal generation.
A. Audio-Haptic Semantic Fusion
Here, we introduce some basic notations utilized in this
section. We define multi-modal dataset as {V, A, H} and the
three modalities involved as V for visual signals, A for audio
signals, and H for haptic signals. Y denotes the category label
set. In the first stage, we try to extract low-level features from
the received audio and haptic signals and produce high-level
feature fusion by semantic constraints.
For the received audio signals, spectrogram is acquired
through spectrum analysis and then deep convolutional neural
network (CNN) is used to obtain hierarchical auditory features.
For the received haptic signals, power spectrum density is used
as the feature descriptors and then the combinations of gated
recurrent unit (GRU) and fully-connected layers are adopted to
obtain haptic features. The specific process can be represented
as follows:
a =Fa (A; θa ) : a ∈ Rda , (1)
h =Fh (H; θh ) : h ∈ Rdh , (2)
where a and h denote low-level features of audio and haptic
signals. Rda and Rdh are their dimensions. θa and θh respec-

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
tively denote parameters of audio mapping network Fa and
haptic mapping network Fh .
Subsequently, we employ a deep semantic network to jointly
fuse audio and haptic features. Here, supervision information
such as category label is utilized to maintain the semantically
discriminative ability for the fused feature f . It can be achieved
by defining a semantic discriminative loss Ldis :
f =Ff (a, h; θf ) : f ∈ Rdf ,
K
1 X
Ldis = fsof tmax (fi , yi ; θi ),
K i=1
where the dimension of the fused feature is Rdf and θf are
parameters of deep semantic network Ff . Besides, fi , yi , and
θi are fused feature, category label, and softmax parameters
belonging to the ith category. fsof tmax (fi , yi ; θi ) is the soft-
max cross entropy loss function. It is noted that the category
label here means which class an audio or a haptic signal
belongs to. In the real cross-modal communication system,
label tagging task is usually performed and loaded at the
sender. When the packet has been received, this information
can be parsed and taken as an important semantic information.
By minimizing the above semantic discriminative loss,
the obtained fused features have strong representation and
semantic discrimination abilities.
B. Modality Irrelevant Latent Space Learning
To bridge the heterogeneous gap among different modalities,
adversarial learning is selected to map the audio-haptic fused
feature and feature of real visual signals into a modality
V − V̂
. (10)
irrelevant latent space. In other words, by exploring this latent
space, the discrepancy between the fused feature and the visual
feature can be reduced as much as possible.
Specifically, we combine a CNN with several fully-
connected layers as the visual mapping network to produce
the associated visual features:
v = Fv (V ; θv ) : v ∈ Rdv ,
where v is Rdv dimensional features of real visual signals
V . θv are parameters of the visual mapping network Fv .
Then, the generative adversarial mechanism is utilized to
learn the latent space, as shown in Fig. 6. Here, we build
a feature discriminator D1 for distinguishing the features,
through which the fused features f can continuously approach
v. In other words, a modality irrelevant latent space can be
explored. According to this principle, the discriminator D1
and fusion network Ff are trained by alternatively minimizing
LD1 in Eq. (6) and LFf in Eq. (7),
LD1 =−Ev∼Pdata(v) [log D1 (v; θd1 )]
−Ef ∼Pdata(f ) [log (1 − D1 (f ; θd1 ))] ,
LFf =−Ef ∼Pdata(f ) [log D1 (f ; θd1 )] + λLdis ,
where λ is the regularization parameter.
The idea of selecting the generative adversarial mechanism
is that when D1 cannot distinguish f and v, it makes the
audio-haptic fused feature gradually approach the feature of
(l)
G (f ; θg(l) ) − Fv(L−l) (V ; θv(L−l) )
,
real visual signals from the aspect of intrinsic semantics. The
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
6
latent space with modality irrelevant characteristics can be
derived.
C. Cross-Modal Visual Signal Generation
After the fused features have learned rich modality ir-
relevant semantic information from real visual signals in
latent space, the final step is to generate the desired visual
(3) signals from the fused features. Here, a generative adversarial
network (GAN) is taken to finish this task. Specifically, in
this GAN, the generator G converts f into visual signals V̂ ,
(4)
and the discriminator D2 distinguishes the real visual signals
V from the generated visual signals V̂ . In other words, this
process mainly indicates that G learns the real visual signal
distribution to confuse D2 , and D2 continuously enhances its
discriminative ability to distinguish false visual signals. The
objective function of this step can be defined as:
LD2 =−EV ∼Pdata(V ) [log D2 (V ; θd2 )]
−EV̂ ∼P [log(1 − D2 (V̂ ; θd2 ))], (8)
data(V̂ )
LG =−EV̂ ∼P [log D2 (V̂ ; θd2 )], (9)
data(V̂ )
where V̂ denotes the generated visual signal. By minimizing
LD2 and LG , G is able to continuously capture the distribution
of V and use it to guide the generation of V̂ .
Besides the discriminative loss LD2 , we should also con-
sider the pixel-wise loss. Here, Euclidean distance is used to
completely match the generated visual signal V̂ with the real
visual signal V , forming the restoration loss. The objective
function of the restoration loss Lrec can be expressed as
Lres =
2
Actually, due to the discrepancy of data distribution-
s between real and generated visual signals in the high-
dimensional pixel space, only considering reconstruction loss
may generate the blur visual signal. To handle this issue, we
employ knowledge distillation as an effective tool for further
(5) improving the quality of generated visual signal. Here, knowl-
edge distillation refers to the separation of key distribution
information from an original model and transfers it to a new
model, aiming at increasing the fine-grained degree of the
generated visual signal. Driven by this, we select the visual
mapping network as distillation model to assist the process of
final visual signal generation through hierarchical knowledge
transferring.
The structure of knowledge distillation is illustrated in Fig.
7. Specifically, during the generation process from the fused
feature f to the desired visual signal V̂ , the intermediate result
(l)
produced by layer l of G can be represented as G(l) (f ; θg ).
(l)
θg is the parameter set of layer l in G. Meanwhile, during
the feature extraction process from V to v, the corresponding
(6) (L−l) (L−l)
intermediate result is Fv (V ; θv ), which can also be
(7) produced by the output of layer l in G. θv
(L−l)
is the parameter
set of layer L − l in Fv . L is the number of layers in both Fv
and G. Based on this, the hierarchical constraint loss of visual
signals is defined as
X
LH =
1
(11)
l
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 7

real visual
visual feature extraction network signal

Fv(L )(V ; v(L ) ) ... Fv(L l )(V ; v(L l ) ) ... Fv(1)(V ; v(1) ) V

LH LH L ... LHl ... LH1

f G (1)( f ; g(1) ) ... G (l )( f ; g(l ) ) ... G (L )( f ; g(L ) ) Vˆ

fused visual generation
... network generated
feature visual signal

Fig. 7: The structure of knowledge distillation for visual signal generation

It can be seen from Fig. 6 that the structures of G and Fv are Algorithm 1 The optimization process of the proposed AH-
symmetrical, for instance, have the same network layers and FVR approach
output dimensions. By this way, the knowledge from multiple Input:
levels of visual features are transferred into layers of G. The multi-modal dataset V, A, H, category label Y
quality and granularity of the generated visual signals are Output:
improved. Finally, the whole objective function of cross-modal the generated visual images V̂
visual signal generation process is Initialization:
network parameters θv , θa , θh , θf , θd1 , θd2 , θg
Lgen = LG + αLres + βLH , (12) hyper-parameters: α, β, λ
where α and β are the hyper-parameters that aim to balance learning rate: µ1 , µ2
the three terms. mini-batch size: N
1: Repeat:
2: h ← Fh (H), a ← Fa (A), v ← Fv (V ), f ← Ff (a; h),
D. Optimization Process V̂ ← G (f )
The whole optimization process of the AHFVR approach 3: update θd1 by SGD for Eq. (6):
can be divided in two main stages. In the first stage, the 4: θd1 ← θd1 − µ2 ∇θd1 LD1
objective functions to be optimized are LD1 and LFf (con- 5: update θa , θh , θv , θf by SGD for Eq. (7):
taining Ldis ). In the second stage, the objective functions to 6: θa ← θa − µ1 ∇θa LFf
be optimized are LD2 and Lgen . 7: θh ← θh − µ1 ∇θh LFf
Specifically, during the optimization, we first initialize all 8: θv ← θv − µ1 ∇θv LFf
network parameters. Then, the fusion network Ff and dis- 9: θf ← θf − µ1 ∇θf LFf
criminator D1 are trained by minimizing the loss function in 10: update θd2 by SGD for Eq. (8):
the first stage. Subsequently, the generator G and discrim- 11: θd2 ← θd2 − µ2 ∇θd2 LD2
inator D2 are trained. Finally, all the training steps above 12: update θg by SGD for Eq.(12):
are repeated until convergence. It is noted that the stochastic 13: θg ← θg − µ2 ∇θg Lgen
gradient descent (SGD) algorithm is adopted to minimize 14: Until convergence.
all the loss functions and upgrade all network parameters.
Different learning rates are set for every network, aiming at
balancing the convergence speed. The specific optimization features a0 and h0 are then fused via semantic constraints.
process is summarized in Algorithm 1. For the condition of Subsequently, the fused feature f 0 is fed into the trained G
convergence, it needs to consider both the outputs of two whose output is the generated visual signal Vˆ0 . For the delayed
discriminators and the reconstruction results. Specifically, after visual signal, when it is successfully received, it can be utilized
each training epoch ends, calculate the outputs of D1 and D2 . to update the AHFVR by Algorithm 1, further enhancing the
If these two values approach 0.5, calculate the Lgen by the accuracy and robustness of the approach.
randomly selected samples for validation. If the variation of
Lgen becomes lower than the pre-set threshold, this algorithm VI. E XPERIMENTAL R ESULTS
is considered as convergence.
When the AHFVR has been trained by the above steps, the A. Dataset
parameters of all the networks are already well optimized. In this section, we evaluate the effectiveness of the proposed
When the subsequent audio and haptic signals have been AHFVR approach based on a standard LMT dataset [33].
received while the desired visual signals are impaired or Generally, The LMT dataset is composed of sound, haptic
delayed, the AHFVR will be activated. Specifically, the audio acceleration, and texture images of 108 different surface
signal A0 and haptic signal H 0 are respectively projected into materials gathered by a three-axis accelerometer, a microphone
feature space through CNN and GRU networks. Both extracted and a camera. According to the material properties, these

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
(a) GAN
(d) ViTac
Fig. 8: The restored visual signals by (a) GAN, (b) WGAN, (c) StackGAN, (d) ViTac, (e) AHFVR(haptic-only), and (f)
AHFVR
surface materials are classified into nine categories: mesh,
stone, glossy, wood, rubber, fiber, foam, foils and paper, textile
and fabric. To avoid repetition, we randomly use 80% data in
each category as the training set, and the remaining 20% as
the testing set.
B. Implementation Details
Network Structure: We adopt CNN to perform feature
extraction for audio and visual modality signals, which con-
tains four convolution layers and three fully-connected layers
(1024 → 128 → 6). The number of convolution kernels is
512, 256, 128, 64. The size of the kernels is 5 × 5. Moreover,
GRU with 256 units is used for the haptic feature extraction.
The fusion network is built as a five-layer fully-connected
neural network (512 → 1024 → 512 → 128 → 64), and the
discriminator D1 is a four-layer fully-connected neural net-
work (512 → 1024 → 512 → 1). The generator G has the
same but the reversed network structure with the visual fea-
ture extraction network. The network of discriminator D2 is
composed of four convolution layers and two fully-connected
layers (1024 → 1). The number of convolution kernels is 512,
256, 128, 64. The size of the kernels is 5 × 5. Except for
softmax layer and sigmoid layer, the other feature layers are
activated by relu function and batchnorm regularization.
Training Details: We first set batch size as 25 in the train-
ing procedure. Category labels are coded as one-hot vector.
Network parameters are learned by optimization described in
Section V-D for all modules with λ = 10−3 , α = 0.1 and
β = 10−5 . In details, the learning rate for the generator is
0.0005 and the discriminator is 0.0001. The training procedure
is adopted at 200 epochs for all the datasets.
C. Compared Approaches
Here, we select the following representative models and
approaches for performance comparison to demonstrate the
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
8
(b) WGAN (c) StackGAN
(e) AHFVR(haptic-only) (f) AHFVR
effectiveness of the proposed AHFVR approach.
GAN [34] is the most basic generative adversarial model,
which can generate visual signals by a procedure of max-min
optimization.
WGAN [35] is a variant of GAN whose training aim is to
minimize the Wasserstein distance.
StackGAN [36] is an another variant of GAN which is used
to generate visual signals conditioned on textual description.
ViTac [37] provides an approach based on the improved
GAN to generate visual signals conditioning on the semantic
features encoded by haptic modalities. It considers the corre-
lation between haptic and visual modalities.
AHFVR(haptic-only) is the AHFVR which removes the
audio modality. In this case, feature fusion component is
discarded and only haptic modality is employed to generate
corresponding visual signals. It can also be utilized to verify
the effectiveness of audio modality and multi-modal fusion.
Moreover, we also conduct some baseline experiments,
including AHFVR(audio-only), without latent space learning,
and without-distillation to estimate the validity of each com-
ponent in the proposed AHFVR approach.
D. Evaluation Metrics
Here, we choose the following metrics for evaluation.
Inception Score is a numerical metric for quantitative
evaluation, measuring the performance of the generated visual
signals. Its calculation formula is
 
IS = exp EV̂ KL(p(ŷ|V̂ )kp(ŷ)) , (13)
where V̂ represents a generated visual signal, and ŷ is the
predicted label through Inception networks. p(ŷ) denotes the
marginal distribution. p(ŷ|V̂ ) denotes the possibility of the
category ŷ the given visual signal V̂ belongs to. The higher
inception scores means the better performance.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 9

FID calculates the distance between real signal and gen-

erated signal in the feature space. We first use the Inception
network to extract features and then apply Gaussian model
to fit the feature space, finally calculate the distance between
the two features. The smaller FID represents the better perfor-
mance.
SSIM is an image quality assessment method, generally
used to measure similarity among a couple of visual signals.
The higher SSIM represents the better performance.
PSNR is a common and widely used objective evaluation
metric for visual signals, defined as:
!
2
(2n − 1)
P SN R = 10 × log10 , (14)
M SE
where M SE is the mean squared error between the generated
and the real visual signals. n represents the number of bits.
The higher PSNR means the better performance.
E. Experimental Results and Analysis
The inception score and FID of the proposed AHFVR and
the other competing approaches are given in Table II. From
this table, we can see that the proposed AHFVR approach
obtains the highest inception score and the smallest FID
score. The reasons lie in two aspects: on the one hand, the
AHFVR has the highest semantic correlations, demonstrating
that the semantic consistency between the received audio-
haptic signals and the generated visual signals has been
explored. On the other hand, it also indirectly reveals that the
AHFVR performs well in both quality and diversity. Moreover,
compared with WGAN and StackGAN, the results of ViTac
and AHFVR(haptic-only) approaches are slightly better. The
reason is that they add a semantic correlation learning module
between non-visual modality and visual modality. What’s
more, in Fig. 8, the fine-grained visual signals generated by
the AHFVR reflect the importance of knowledge distillation.
Finally, compared with AHFVR(haptic-only) approach, the
combination and fusion of audio and haptic modalities in
the AHFVR approach is highly significant for accuracy and
robustness.
TABLE II: Inception score and FID of the proposed AHFVR
and the other competing approaches
Approach Inception score
Original 10.25 ± 0.65
GAN 7.73 ± 0.46
WGAN 7.23 ± 0.77
StackGAN 8.75 ± 0.87
ViTac 7.36 ± 0.64
AHFVR(haptic-only) 8.84 ± 0.79
AHFVR 9.88 ± 0.84
Furthermore, the ablation experiment are also performed on
the LMT dataset to verify the effectiveness of components
in the proposed AHFVR approach. Several baseline schemes
are formed by discarding different components in AHFVR
during training phase. We calculate the SSIM metric every
50 epoches. The results are presented in Fig. 9, which are
analyzed as follows:
Fig. 9: SSIM performance of the approaches in the ablation
experiment
Only Haptic/Audio Modality for Fusion: This scheme
only employs haptic or audio signals as prior knowledge to
train the AHFVR. From Fig. 9 we can obviously see that the
results are excessively unsatisfied. In other words, it is difficult
to use only haptic or audio modality signals to generate visual
signals with explicit content.
Without Latent Space Learning: This scheme cannot
perform the modality irrelevant latent space learning in Section
V-B. Comparing with the AHFVR approach, this scheme
achieves worse performance. Thus, it is evident that latent
space learning can offer more accurate and detailed visual
elements from the real visual signals in training dataset.
Without Distillation: This scheme generates visual signals
without the guidance of real image in hierarchical latent space.
In other words, the knowledge distillation is not considered.
We can clearly observe from Fig. 9 that the quality of the
generated visual signals is poor. Therefore, knowledge distil-
lation technique can help the AHFVR achieve more desirable
performance.
VII. P RACTICAL P LATFORM FOR E VALUATING THE
C ROSS -M ODAL S IGNAL R ECONSTRUCTION S TRATEGY
In this section, a practical cross-modal communication plat-
FID form is constructed for evaluating the proposed signal recon-
36.97 struction strategy. The structure of this platform is depicted in
134.12
142.06
Fig. 10, which is composed of the master domain, the network
122.34 domain, and the slave domain. Among them, the master
125.20 domain is a haptic device connected to a laptop. The haptic
111.48 device is used to send control commands and receive force
48.84
feedback, and the laptop is used to display the received images
and audio. The slave domain is the combination of a robot
arm equipped with force sensors, a microphone, and a camera
for the collection of haptic, audio, and visual signals. For the
network domain, multi-modal streams are delivered via the
public 5G network and edge servers. All of the received signals
belong to nine semantic categories: slate, wood, cardboard,
silk, foam, brass, linen, bubble cushion film, porous plastic
sheet.

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 10

base station

Micro-
phone

gateway
camera force sensors
database
router
Master Domain Network domain Slave Domain

Fig. 10: Schematic overview of the practical cross-modal communication platform

(a) GAN (b) WGAN (c) StackGAN

(d) ViTac (e) AHFVR(haptic-only) (f) AHFVR

Fig. 11: The restored visual signals by (a) GAN, (b) WGAN, (c) StackGAN, (d) ViTac, (e) AVHR(haptic-only), and (f)
cross-modal signal construction with AHFVR

Fig. 11 shows those reconstructed visual signals and provide TABLE III: SSIM and PSNR of the proposed cross-modal
some quantitative information about generation quality. In signal reconstruction with AHFVR and the other competing
particular, SSIM and PSNR are selected to measure similarity schemes
between the reconstructed visual signals and real visual signal-
Approach SSIM PSNR
s. The results are given in Table III. It can be seen that the AH- Original 0.950 −
FVR achieves the best performance on the practical platform. GAN 0.199 9.782
Moreover, the basic generative models, like GAN and WGAN, WGAN 0.201 10.056
always achieve the worst performance. Although StackGAN StackGAN 0.216 10.178
ViTac 0.235 10.678
has made some improvements, it is still limited by discrep- AHFVR(haptic-only) 0.241 10.831
ancy of heterogeneous streams. Compared with AHFVR, the AHFVR 0.412 14.021
performance of AHFVR(haptic-only) is inferior due to the
lack of supplementary audio information. In summary, the
proposed perception-aware cross-modal signal reconstruction They give their experience scores 1∼5 from two aspects: qual-
strategy obtains the best performance, guaranteeing quality of ity scale and classification scale (discrimination ability of the
the received signals. reconstructed images). They respectively represent quality and
Finally, in order to verify the performance of the proposed discrimination quality of the reconstructed images compared
strategy, user’s quality of experience (QoE) is measured based with those of the real images. Scores from the participants
on their subjective assessment of the synthesized images. are averaged and the results are shown in Fig. 12. It can be
Therefore, in this qualitative evaluation, 15 participants who observed that the proposed cross-modal signal reconstruction
have normal senses are invited to manipulate the haptic devices strategy with the AHFVR achieves the best performance on
at the master terminal and perceive the received visual signals. end user’s experience.

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309
IEEE TRANSACTIONS ON MULTIMEDIA
AHFVR
(haptic-only)
Fig. 12: QoE performance of all the approaches on the
practical cross-modal communication platform
VIII. C ONCLUSION
In this paper, we propose cross-modal signal reconstruction
from the perspective of human’s perceptual facts. The key
insight is that it borrows the related mechanisms from the
human’s perception system, which has several similarities with
cross-modal communications. As an initial trial, we design a
time-frequency masking-based audio-haptic redundancy elim-
ination mechanism as well as an audio-haptic fused visual
signal restoration approach. The ultimate aim is to guarantee
quality of received visual signals and end-user’s experience.
From a standard multi-modal dataset and a practical cross-
modal communication platform, the effectiveness of the pro-
posed architecture and strategy can be verified.
The present paper also leaves several open and interest-
ing questions. First, what’s the theoretical threshold about
restoration of one modality signals from the other modality
signals. Second, during transmission in several scenarios, the
impaired or delayed phenomenon of visual, audio, haptic
signals may randomly occur. How to effectively handle this
issue when more than one modality signal is in trouble. Third,
the perceptual facts and theory adopted in this paper is rela-
tively superficial. More thorough analysis and exploitation for
characteristics and interaction mechanisms of human’s visual,
auditory, haptic sensory systems are needed for designing
more powerful cross-modal transmission and signal processing
approaches. We will carefully study them in our ongoing
works.
R EFERENCES
[1] Z. Yuan, G. Ghinea, and G-M. Muntean, “Beyond multimedia adap-
tion: Quality of experience-aware multi-sensorial media delivery,” IEEE
Trans. Multimedia, vol. 17, no. 1, pp. 104-117, 2015.
[2] W. Zhu, X. Wang, and W. Gao, “Multimedia intelligence: When mul-
timedia meets artificial intelligence,” IEEE Trans. Multimedia, vol. 22,
no. 7, pp. 1823-1835, 2020.
[3] L. Zhou, D. Wu, X. Wei, and J. Chen, “Cross-modal stream scheduling
for ehealth,” IEEE J. Sel. Area Comm., vol. 39, no. 2, pp. 426-437, 2021.
[4] L. Zhou, D. Wu, J. Chen, and X. Wei, “Cross-modal collaborative
communications,” IEEE Wirel. Commun., vol. 27, no. 2, pp. 112-117,
2020.
© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.
11
[5] X. Wei and L. Zhou, “AI-enabled cross-modal communications,” IEEE
Wirel. Commun., vol. 28, no. 4, pp. 182-189, 2021.
[6] B. Cizmeci, X. Xu, R. Chaudhari, N. Alt, and E. Steinbach, “A
multiplexing scheme for multimodal teleoperation,” ACM Trans. Multim.
Comput., vol. 13, no. 2, p. 21, 2017.
[7] M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, “Toward low-
latency and ultra-reliable virtual reality,” IEEE Network, vol. 32, no. 2,
pp. 78-84, 2018.
[8] X. Wei, M. Zhang, and L. Zhou, “Cross-modal transmission strategy,”
IEEE Trans. Circ. Syst. Vid., vol. 32, no. 6, pp. 3991-4003, 2022.
[9] H. Cheng, Y. Guo, J. Yin, H. Chen, J. Wang, and H. Nie, “Audio-
driven talking video frame restoration,” IEEE Trans. Multimedia, DOI:
10.1109/TMM.2021.3118287, 2021.
[10] S. Zou, Q. Wang, J. Ge, and Y. Tian, “Peer-assisted video streaming with
RTMFP flash player: A measurement study on PPTV,” IEEE Trans. Circ.
Syst. Vid., vol. 28, no. 1, pp. 158-170, 2018.
[11] M. T. Hossan, M. Z. Chowdhury, M. Shahjalal, and Y. M. Jang, “Human
bond communication with head-mounted displays: Scope, challenges,
solutions, and applications,” IEEE Commun. Mag., vol. 57, no. 2, pp.
26-32, 2019.
[12] X. W. Tang, X. L. Huang, F. Hu, and Q. Shi, “Human-perception-
oriented pseudo analog video transmissions with deep learning,” IEEE
Trans. Circ. Syst. Vid., vol. 69, no. 9, pp. 9896-9909, 2020.
[13] X. Yi, K. Yang, X. Zhao, Y. Wang, and H. Yu, “AHCM: Adaptive
huffman code mapping for audio steganography based on psychoacoustic
model,” IEEE Trans. Inf. Foren. Sec., vol. 14, no. 8, pp. 2217-2231, 2019.
[14] R. Chaudhari, C. Schuwerk, M. Danaei, and E. Steinbach, “Perceptual
and bitrate-scalable coding of haptic surface texture signals,” IEEE J.
Sel. Top. Signal Process., vol. 9, no. 3, pp. 462-473, 2015.
[15] B. Li, J. P. Munoz, X. Rong, Q. Chen, and J. Xiao, “Vision-based mobile
indoor assistive navigation aid for blind people,” IEEE Trans. Mobile
Comput., vol. 18, no. 3, pp. 702-714, 2019.
[16] M. Yuan and Y. Peng, “CKD: Cross-task knowledge distillation for text-
to-image synthesis,” IEEE Trans. Multimedia, vol. 22, no. 8, pp. 1955-
1968, 2020.
[17] H. Liu, Y. Wu, F. Sun, B. Fang, and D. Guo, “Surface material retrieval
using weakly paired cross-modal learning,” IEEE Trans. Autom. Sci.
Eng., vol. 16, no. 2, pp. 781-791, 2019.
[18] M. S. Siddiqui, S. O. Amin, and C. S. Hong, “A set-top box for end-to-
end QoS management and home network gateway in IMS,” IEEE Trans.
Consum. Electr., vol. 55, no. 2, pp. 527-534, 2009.
[19] E. Mohamad and A. E. Saddik, “Admux communication protrocol for
real-time multimodal intreaction,” IEEE/ACM Int. Sym. Distr. Simulat.
Real Time Appl., Dublin, Ireland, pp. 118-123, 2012.
[20] L. Madmoni, S. Tibor, I. Nelken, and B. Rafaely, “The effect of
partial time-frequency masking of the direct sound on the perception of
reverberant speech,” IEEE-ACM Trans. Audio Spe., vol. 29, pp. 2037-
2047, 2021.
[21] R. Hassen, B. Gulecyuz, and E. Steinbach, “PVC-SLP: Perceptual
vibrotactile-signal compression based-on sparse linear prediction,” IEEE
Trans. Multimedia, vol. 23, pp. 4455-4468, 2021.
[22] J. Yang, C. Wang, B. Jiang, H. Song, and Q. Meng, “Visual perception
enabled industry intelligence: State of the art, challenges and prospects,”
IEEE Transactions on Industrial Informatics, vol. 17, no. 3, pp. 2204-
2219, 2021.
[23] J. Yin, B. Chen, and Y. Li, “Highly accurate image reconstruction
for multimodal noise suppression using semisupervised learning on big
data,” IEEE Trans. Multimedia, vol. 20, no. 11, pp. 3045-3056, 2018.
[24] G. Yuan and B. Ghanem, “`0 TV: A Sparse optimization method for
impulse noise image restoration,” IEEE Trans. Pattern Anal., vol. 41,
no. 2, pp. 352-364, 2019.
[25] Z. Yang, H. Liu, and D. Cai, “On the diversity of conditional image
synthesis with semantic layouts,” IEEE Trans. Image Process., vol. 28,
no. 6, pp. 2898-2907, 2019.
[26] H. Xie, A. Boukerche, and A. A. F. Loureiro, “MERVS: A novel
multichannel error recovery video streaming protocol for vehicle ad hoc
networks,” IEEE Trans. Veh. Technol., vol. 65, no. 1, pp. 923-935, 2016.
[27] T. Lin, H. Tseng, Y. Wen, F. Lai, C. Lin, and C. Wang, “Reconstruction
algorithm for lost frame of multiview videos in wireless multimedia sen-
sor network based on deep learning multilayer perceptron regression,”
IEEE Sens. J., vol. 18, no. 23, pp. 9792-9801, 2018.
[28] Y. Peng, K. Cao, and P. C. Cosman, “Generalization of the dark channel
prior for single image restoration,” IEEE Trans. Image Process., vol. 27,
no. 6, pp. 2856-2868, 2018.
[29] Z. Jin, M. Z. Iqbal, D. Bobkov, W. Zou, X. Li, and E. Steinbach,
“A flexible deep CNN framework for image restoration,” IEEE Trans.
Multimedia, vol. 22, no. 4, pp. 1055-1068, 2020.
 This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TMM.2022.3194309

IEEE TRANSACTIONS ON MULTIMEDIA 12

[30] S. Niu, Y. Jiang, and B. Chen, “Cross-modality transfer learning for Liang Zhou (SM’18) received his Ph.D. degree
image-text information management,” ACM Transactions on Manage- major at electronic engineering both from École
ment Information Systems, vol. 13, no. 1, p. 14, 2022. Normale Supérieure (E.N.S.), Cachan, France and
[31] R. M. Chen, N. N. Mahatme, Z. J. Diggins, and L. Wang, “Impact of Shanghai Jiao Tong University, Shanghai, China in
temporal masking of flip-flop upsets on soft error rates of sequential 2009. Now, he is a professor in Nanjing Univer-
circuits,” IEEE Trans. Nucl. Sci., vol. 64, no. 8, pp. 2098-2106, 2017. sity of Posts and Telecommunications, China. His
[32] N. Otsu, “A threshold selection method from gray-level histograms,” research interests are in the area of multimedia
IEEE Transactions on Syst, Man, Cybernetics, vol. 9, no. 1, pp. 62-66, communications and computing.
1979.
[33] M. Strese, C. Schuwerk, A. Iepure, and E. Steinbach, “Multimodal
feature-based surface material classification,” IEEE Trans. Haptics, vol.
10, no. 2, pp. 226-239, 2017.
[34] C. Wang, C. Xu, X. Yao, and D. Tao, “Evolutionary generative adversar-
ial networks,” IEEE Trans. Evolut. Comput., vol. 23, no. 6, pp. 921-934,
2019.
[35] I. Deshpande, Y. Hu, R. Sun, A. Pyrros, and A. Schwing, “Max-sliced
Wasserstein distance and its use for GANs,” IEEE/CVF Conf. Comput.
Vis. Pattern Recog. (CVPR), Long Beach, USA, pp. 10640-10648, 2019.
[36] H. Zhang, T. Xu, H. Li, and S. Zhang, “StackGAN++: Realistic image
synthesis with stacked generative adversarial networks,” IEEE Trans.
Pattern Anal., vol. 41, no. 8, pp. 1947-1962, 2019.
[37] J. Lee, D. Bollegala, and S. Luo, “Touching to see and seeing to
feel: Robotic cross-modal sensory data generation for visual-tactile
perception,” Int. Conf. Robot. Autom. (ICRA), Montreal, Canada, pp.
4276-4282, 2019.

Xin Wei (M’19) received his Ph.D. degree major

at information and communication engineering from
Southeast University, Nanjing, China in 2009. Now,
he is a professor in Nanjing University of Posts and
Telecommunications, China. His research interests
are in the areas of multimedia communications and
computing, educational technology.

Yuyuan Yao received her M.S. degree major at

information and communication engineering from
Nanjing University of Posts and Telecommunica-
tions, Nanjing, China in 2022. Her research interest
is in the area of multimedia communications.

Haoyu Wang received his B.E. degree major at

communication engineering from Nanjing Universi-
ty of Posts and Telecommunications, Nanjing, China
in 2021, where he is currently pursuing the M.S.
degree major at information and communication
engineering. His research interest is in the area of
multimedia communications.

© 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Macau Univ of Science and Technology. Downloaded on January 15,2023 at 18:45:21 UTC from IEEE Xplore. Restrictions apply.