See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/329355334

Decoding natural scenes based on sounds of objects within scenes using

multivariate pattern analysis

Article  in  Neuroscience Research · December 2018

DOI: 10.1016/j.neures.2018.11.009

CITATIONS READS

0 24

7 authors, including:

Junhai Xu Xianglin Li
Tianjin University 24 PUBLICATIONS   71 CITATIONS   
29 PUBLICATIONS   176 CITATIONS   
SEE PROFILE
SEE PROFILE

Baolin Liu
Tianjin University
46 PUBLICATIONS   316 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Medical image analysis View project

Neural mechanism of affective processing View project

All content following this page was uploaded by Junhai Xu on 15 April 2019.

The user has requested enhancement of the downloaded file.

G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
Neuroscience Research xxx (2018) xxx–xxx

Contents lists available at ScienceDirect

Neuroscience Research
journal homepage: www.elsevier.com/locate/neures

Decoding natural scenes based on sounds of objects within scenes

using multivariate pattern analysis
Xiaojing Wang a,1 , Jin Gu a,1 , Junhai Xu a , Xianglin Li c , Junzu Geng d , Bin Wang c ,
Baolin Liu a,b,∗
a
College of Intelligence and Computing, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, 300350, PR China
b
State Key Laboratory of Intelligent Technology and Systems, National Laboratory for Information Science and Technology, Tsinghua University, Beijing,
100084, PR China
c
Medical Imaging Research Institute, Binzhou Medical University, Yantai, Shandong 264003, PR China
d
Department of Radiology, Yantai Affiliated Hospital of Binzhou Medical University, Yantai, Shandong, 264003, PR China

a r t i c l e i n f o a b s t r a c t

Article history: Scene recognition plays an important role in spatial navigation and scene classification. It remains
Received 27 March 2018 unknown whether the occipitotemporal cortex could represent the semantic association between the
Received in revised form scenes and sounds of objects within the scenes. In this study, we used the functional magnetic resonance
21 November 2018
imaging (fMRI) technique and multivariate pattern analysis to assess whether different scenes could be
Accepted 30 November 2018
Available online xxx
discriminated based on the patterns evoked by sounds of objects within the scenes. We found that pat-
terns evoked by scenes could be predicted with patterns evoked by sounds of objects within the scenes in
the posterior fusiform area (pF), lateral occipital area (LO) and superior temporal sulcus (STS). The further
Keywords:
Cross modality functional connectivity analysis suggested significant correlations between pF, LO and parahippocampal
Scene decoding place area (PPA) except that between STS and other three regions under the scene and sound conditions.
Functional connectivity A distinct network in processing scenes and sounds was discovered using a seed-to-voxel analysis with
Multivariate pattern analysis STS as the seed. This study may provide a cross-modal channel of scene decoding through the sounds
fMRI of objects within the scenes in the occipitotemporal cortex, which could complement the single-modal
channel of scene decoding based on the global scene properties or objects within the scenes.
© 2018 Elsevier B.V. and Japan Neuroscience Society. All rights reserved.

1. Introduction PPA has been shown to be selectively activated by the scene

Scene recognition is one of the main aspects in perceiving
the world, and people could recognize the environment around
us with a high speed and accuracy through scene recognition,
which plays a central role in navigating to a destination and scene
discrimination. Multiple areas have been proved to be engaged
in the understanding of scene recognition. Neuroimaging studies
have demonstrated that the brain regions were recruited in scene
perception mainly including parahippocampal place area (PPA)
(Epstein and Kanwisher, 1998; Epstein et al., 2007; Walther et al.,
2011) and lateral occipital complex (LOC) (Malach et al., 1995; Park
et al., 2011, 2015).
∗ Corresponding author at: College of Intelligence and Computing, Tianjin Key Lab-
oratory of Cognitive Computing and Application, Tianjin University, Tianjin, 300350,
PR China.
E-mail address: liubaolin@tsinghua.edu.cn (B. Liu).
1
Xiaojing Wang and Jin Gu contribute equally to this work.
than other visual stimuli, and especially to the spatial layout of the
scene, suggesting that PPA could represent the scene by encoding
the spatial geometry of local environments (Epstein and Kanwisher,
1998). Further evidence suggested that PPA made a more sensi-
tive response to the indoor than outdoor scenes (Henderson et al.,
2007). Another study (Walther et al., 2009) found that activities in
PPA and LOC included the classification information of the natural
scenes. As previous work suggests that the two subregions of LOC,
the pF and the LO, may support different functions during visual
recognition (Haushofer et al., 2008; Drucker and Aguirre, 2009). By
exploring the two subregions of the LOC: lateral occipital area (LO)
and posterior fusiform area (pF), MacEvoy and his colleagues sug-
gested the existence of the neural mechanism that patterns evoked
by scenes could be predicted by the patterns evoked by within-
scene objects in LO (MacEvoy and Epstein, 2011). Furthermore, by
classifying the patterns of the scene based on the object patterns,
some recent studies indicated that the regions which mediated the
within-scene object information were not limited to LOC, maybe
including PPA (Harel et al., 2013; Linsley and MacEvoy, 2014).

https://doi.org/10.1016/j.neures.2018.11.009
0168-0102/© 2018 Elsevier B.V. and Japan Neuroscience Society. All rights reserved.

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
2 X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx
Previous studies found that superior temporal sulcus (STS) could
not only process the visual and auditory information from the ani-
mals and man-made manipulable objects (tools), but also integrate
the audio-visual information (Beauchamp et al., 2004; Tyll et al.,
2013; Venezia et al., 2017), which provided support to the view
that it was a nature of our human brain to integrate the relevant
information from multiple modalities (Mesulam, 1998; Liang et al.,
2013). Recently, the STS has been proposed to be quite active and
responsible for different processing mechanisms in distinct tasks
(Hein and Knight, 2008). Accordingly, we speculate that STS may
also be involved in processing the semantic relationship between
scenes and object sounds.
Information integration across different sensory modalities con-
tributes to object recognition (Beauchamp, 2005; Doehrmann et al.,
2010). Visual and auditory information in objects could activate
the modality-specific brain regions, which implied that multisen-
sory convergence zones were not fixed but rather depended on
object contents and modalities (Amedi et al., 2005). One recent
study (Vetter et al., 2014) found that the early visual cortex could
distinguish the perception from the imagery contents, probably
because the actual sound stimuli induced people to imagine the
corresponding category information, and this finding provided a
support to the speculation that a perceptual integration mecha-
nism appeared in the human primary cortex besides the advanced
cortex (Werner and Noppeney, 2010; Klemen and Chambers, 2012;
Rohe and Noppeney, 2016).
Previous studies have suggested that there are two main chan-
nels in scene recognition: the spatial property-based channel
(Renninger and Malik, 2004; Greene and Oliva, 2009) and object-
based channel (MacEvoy and Epstein, 2011; Stansbury et al., 2013),
and these two channels were complementary to each other. The
finding that PPA and LOC could represent the scene in the dis-
tributed and complemented way supported the evidence that the
spatial layout and scene content representation were processed in
different channels (Park et al., 2011). In addition, LO was proposed
to provide a new object-based channel to scene decoding, which
complemented the processing of the spatial attributes in the PPA
(MacEvoy and Epstein, 2011). The natural scene recognition was
not only tuned by the objects within the scene but also influenced
by the global properties (Greene and Oliva, 2009). However, these
channels were confined in the visual modality. It remains unclear
whether the other modalities such as the scene-relevant typical
sounds are beneficial to scene recognition.
The multivariate pattern analysis (MVPA) is capable of quanti-
fying the activity patterns of each item within the category and is
very sensitive to different categories (Haxby et al., 2001; Harrison
and Tong, 2009). To our knowledge, no study to date has inves-
tigated to what extent the areas that are related to the visual
scenes, objects and audio-visual integration are involved in rep-
resenting the semantic relationship between the visual scenes and
their sounds using MVPA. In the present study, we hypothesized
that the sounds most closely linked to the objects within the scene
could decode this scene. To test the hypothesis, we used the func-
tional magnetic resonance imaging (fMRI) technique to acquire
blood oxygenation level dependent (BOLD) data while participants
viewed the four categories of scenes (indoor scenes vs outdoor
scenes) and listened to eight categories of sounds (two sounds per
scene). Four regions of interest (ROIs) were defined and then MVPA
was performed using the voxel-wise activity patterns of each stim-
ulus to examine whether the patterns for the sounds could decode
the scene patterns in all four ROIs. To explore the influence of the
scene openness, we divided the scenes into the indoor and outdoor
scenes and made the same MVPA analysis. Finally, the functional
connectivity (FC) analyses between all four ROIs were made to
examine the functional integrations in the scene and sound tasks.
Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
2. Material and methods
2.1. Participants
Twenty-three healthy subjects (all right-handed, 12 females,
average: 21.91 ± 2.81 years old, ranged from 18 to 26 years old)
participated in the study, in which 2 subjects were excluded from
further analyses due to the excessive head movement during scan-
ning, thus, a total of 21 effective subjects participated in the present
study. All participants had no history of neurological, psychiatric
diseases, or auditory impairments, and had a self-reported normal
or correct-to-normal hearing and vision. Written informed con-
sents were obtained from all participants before the experiment,
and the study was approved by the Institutional Review Board (IRB)
of Tianjin Key Laboratory of Cognitive Computing and Application,
Tianjin University.
2.2. Experiment stimuli
All stimuli consisted of 32 color images of scenes and 64 sound
clips of animals or man-made manipulable objects (tools) selected
from the internet There were four categories of scenes (indoor:
kitchen and office; outdoor: street and grass) and the sample
images were shown in Fig. 1A. The scene images were edited to
400 * 400 pixels with Adobe Photoshop with the same parame-
ters, trying to avoid irrelevant factors. We chose eight categories
of sound clips strongly associated with the objects in the scenes
(“vroom” for the engine and “hoot” for the horn in the street, “siz-
zle” for the hot oil and “rat-tat” for the kitchen knife in the kitchen,
“moo” for the cattle and “baa” for the sheep in the grass, “click” for
the keyboard and “ringing” for the telephone in the office), and each
category included eight sounds clips. To reduce the possible con-
founds evoked by vocalizations (Belin et al., 2000; Norman et al.,
2006), all the animal and tool sounds did not contain any vocaliza-
tions or vocal-related content at all. All sound stimuli were edited to
2.5-s duration and converted to one channel (mono, 44.1KHZ, 16-
bit), 80–83 dB C-weighted in both ears (Cool Edit Pro, Syntrillium
Software Co., owned by Adobe). Sounds were presented to subjects
binaurally. All stimuli were assessed by another 10 volunteers to
make sure their easy recognition (average accuracy = 98.78%, stan-
dard deviation = 0.014).
2.3. Experimental design
A block design was adopted in the experiment (Fig. 1B). Scan
sessions consisted of 4 experimental runs, and each run lasted for
7 min 46 s which comprised 12 blocks (4 scene blocks, 8 sound
blocks) that were presented in a random order. Each block con-
sisted of 8 different stimuli, in which one 2.5-s trial followed by
0.5-s inter-stimulus interval (ISI) was repeated 8 times with differ-
ent stimuli. Following each block, one 4-s “select one from four”
button was pressed by each subject to record which category of the
scene the stimuli they had seen or heard belonged to. A 10-s white
fixation with the gray background was followed. Before the fMRI
scan, subjects received a train to learn about stimuli categories by
some exemplar stimuli. In addition, this training can help assure
subjects could identify scene pictures and object sounds properly.
Subjects were asked to silently name the item when each scene
picture or sound clip appeared (e.g. When heard the “vroom”, one
should name it “sound of the engine” silently) and all the blocks as
well as trials in each block were presented randomly across runs
with the same stimuli as in the first run.
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx 3

Fig. 1. Experimental materials and paradigm. (A) Experimental materials. The sample pictures correspond to 4 kinds of scenes (office, grass, street and kitchen) and there are
32 scene images in total. The speaker icons represents the objects sounds from 4 categories scene, and there are 8 kinds sounds in total (“vroom” for the engine and “hoot”
for the horn in the street, “sizzle” for the hot oil and “rat-tat” for the kitchen knife in the kitchen, “moo” for the cattle and “baa” for the sheep in the grass, “click” for the
keyboard and “ringing” for the telephone in the office). (B) Block-design paradigm. The experiment was composed of 4 runs and lasted about 29 min. Two task blocks were
separated by 10-s blocks of rest, and each task block contained 8 stimuli (shown for 2.5 s with 0.5-s inter-stimulus interval) presented centrally, with a following 4-s button
press.

2.4. Data acquisition
The experiment scanning was conducted on a 3.0 T Siemens
skyra scanner with a 20-channel head coil at Yantai Affili-
ated Hospital of Binzhou Medical University. Foam pads and
earplugs were used to reduce the head motion and scanner
noise. A high-resolution structural MR image set was collected
using a T1 weighted 3D MPRAGE sequence (repetition time
(TR) = 1900 ms, echo time (TE) = 2.52 ms, voxel size = 1 × 1 × 1
mm3 , matrix size = 256 × 256, flip angle (FA) = 9◦ ). A gradient-echo
planar imaging (EPI) sequence (TR = 2000 ms, TE = 30 ms, voxel
size = 3.1 × 3.1 × 4.0 mm3 , matrix size = 64 × 64, slices = 33, slices
thickness = 4 mm, slices gap = 0.6 mm, FA = 90◦ ) was used for func-
tional data collection. The stimuli presentation and behavioral
response collection were performed by E-Prime 2.0 Profes-
sional (Psychology Software Tools, Pittsburgh, PA, USA) through
the audio-visual somatosensory device equipment with high-
resolution glasses and headphones.
2.5. Data preprocessing
Data preprocessing was conducted using the SPM8 package
(http://www.fil.ion.ucl.ac.uk/spm). For reaching steady-state equi-
librium, we discarded the first five functional images from each
run, and the remaining images were slice time-corrected to the first
image of the first run and motion-corrected by a realign analysis.
For each participant, the individual’s own structural image was first
coregistered to the mean functional image after motion correction,
and the transformed structural image was then segmented into
gray matter, white matter and cerebral spinal fluid (CSF) by using
a unified segmentation algorithm. And the corrected images were
spatially normalized to 3 × 3 × 3 mm3 in the Montreal Neurological
Institute (MNI) space using the generated spatial parameters from
the segment analysis. For each individual subject, the functional
data were spatially smoothed with a full-width at half maximum
(FWHM) Gaussian 6 × 6 × 6 mm kernel. It is noted that data for the
classification analysis were not smoothed.
The fMRI data for each subject were preprocessed to remove the
low-frequency signal changes and minimize the head move arti-
facts. Then a general linear model (GLM) was constructed for the
smoothed functional volumes in all runs to obtain the voxel-wise
responses (␤ values) corresponding to each condition.
2.6. ROI definition
First of all, we obtained the activated brain regions by all
experimental conditions versus rest at the group level, and
then picked the anatomical masks in the AAL atlas (Lalli et al.,
2012) (Fusiform L, Fusiform R, Occipital Mid L, Occipital Mid R,
ParaHippocampal L, ParaHippocampal R, Temporal Sup L, Tem-
poral Sup R) corresponding to pF, LO, PPA and STS with the
WFU Pickatlas toolbox in SPM8. Finally, four ROIs (Fig. 2) were
defined based on the common parts between the activated regions
(p < 0.05, FDR corrected) and the anatomical masks.
2.7. Data analysis
2.7.1. Univariate analysis
A univariate analysis was performed to quantify the percent sig-
nal change of the sound and scene conditions for each ROI. At first,
we used the MarsBar toolbox (http://marsbar.sourceforge.net) to
extract the time courses for four ROIs in each condition, and then
calculated the average of the signal change for each region in the
sound and scene conditions separately. Finally, the paired t-tests
were conducted between the signal changes of the sound and the
scene in each region to investigate whether the activation are dif-
ferent between sound and scene conditions.
2.7.2. Classification analysis using MVPA
MVPA was conducted to explore the relationship between the
scene and the associated sounds in all ROIs. We added labels to
the response patterns of sound according to the scene categories
(kitchen, grass, office, street), for example, the patterns of the
engine sound labeled as “street”, and the response patterns of scene
were labeled same as the sound. And then a linear SVM classifier
was chosen to classify patterns evoked by scene images based on
the patterns evoked by sounds. The LibSVM toolkit was used to
implement the four-way classification in the ROIs (http://www.
csie.ntu.edu.tw/∼cjlin/libsvm/). Afterward, the one-sample t-test
was conducted on the classification performances to test whether
the statistical value was statistically significant (p < 0.05). In order

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
4 X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx

Fig. 2. Regions of interest. pF: posterior fusiform area; LO: lateral occipital area; PPA: parahippocampal place area; STS: superior temporal sulcus.

to assure the classification performances were reliable, we shuf- Table 1

Reaction time and accuracies of all conditions.
fled the labels and randomly assigned to the training samples and
performed the 4 scene classification analysis with the same proce- Conditions Response time (ms) Response accuracy (%)
dure based on the “shuffled” data (Stelzer et al., 2013). Furthermore, Sounds Cattle 1472.81 ± 115.12 98.62 ± 0.12
we divided the scene categories into indoor and outdoor scenes Sheep 1303.93 ± 100.52 100.00 ± 0.00
and performed a paired classification analysis, and the paired two- Wok 1239.91 ± 134.22 95.82 ± 0.12
tailed t-test was used to examine the statistical distinction of the Kitchen knife 1226.52 ± 116.92 93.12 ± 0.21
Horn 1374.02 ± 121.812 97.22 ± 0.11
classification results of the scene sub-categories.
Engine 1540.22 ± 116.52 90.32 ± 0.21
In addition, we performed the within-cue classifications with Telephone 1298.42 ± 84.52 100.00 ± 0.00
the same method. The discrimination analysis was conducted on Keyboard 1262.72 ± 87.73 84.73 ± 0.32
the four kinds of scene based on the patterns evoked by scene Pictures Grass 1269.52 ± 122.82 100.00 ± 0.00
Kitchen 1120.43 ± 102.31 95.83 ± 0.21
images, and eight object sounds using the patterns evoked by
Street 1532.93 ± 136.11 97.22 ± 0.14
sounds. Office 1205.71 ± 97.72 98.62 ± 0.11
To eliminate the interference of the voxel numbers on the results
(Walther et al., 2009; Said et al., 2010), we repeated the proce-
dure of the classification analysis by controlling the voxel numbers
using the F score method (Chen and Lin, 2006). F-score is a simple
technique which can measure the discrimination of a feature. We
calculated F-score of each voxel and then selected specified num-
ber of voxels (from 50 to 950) according to F-scores in a descending
order, and then applied the SVM classifier to conduct the classi-
fication procedure. ANOVA was conducted to test whether there
were significant differences in the classification accuracies when
the voxel numbers increased in each ROI.

2.7.3. Functional connectivity analysis

Finally, we performed a ROI-to-ROI FC analysis using pF, LO, PPA
and STS as seeds to investigate the functional coordination between
four ROIs in the sound and scene conditions. The preprocessed data
in SPM8 were imported into FC Toolbox conn (Whitfield-Gabrieli
et al., 2012) (http://www.nitrc.org/projects/conn). Individual nor-
Fig. 3. The average of signal change of sound and scene in all ROIs. The signal change
malized anatomical images were segmented into white matter, was calculated using Marsbar toolbox and paired t-test was performed between the
gray matter and CSF. The confounding factors including possi- sound and corresponding scene in each ROI. The error bars denoted the standard
ble motion, physiological and other human factors were defined error of the mean. *p < 0.05, **p < 0.01, ***p < 0.001.
as the principle components of the BOLD signals from the white
matter and CSF, without regressing out the mean global brain sig-
tively. Table 1 showed the response time and accuracies in all
nal. Pearson’s correlation coefficients were calculated between the
conditions.
time-courses of the seed region and all other voxels.

3.2. Univariate analysis

3. Results
3.1. Behavioral performance
Subjects were asked to press the button accordingly after a block
of stimuli and silently name every item during scanner to control
the subjects’ attention, and their behavioral performances were
evaluated by calculating the reaction time of button pressing. The
mean accuracy was 95.90% ± 0.046 (t(11) = 72.18, p < 0.001) and
the mean response time was 1325.26 ± 138.51 ms (t(11) = 31.73,
p < 0.001), indicating that subjects could identify the stimuli effec-
The average of signal changes of all the sounds and scenes was
calculated separately using the Marsbar package, as shown in Fig. 3.
A one-sample t-test was performed for each condition in the four
regions and all were significantly above the chance level both in
the sound condition (pF: 0.09%, t(20) = 8.58, p < 0.001; LO: 0.11%,
t(20) = 9.61, p < 0.001; PPA; 0.04%, t(20) = 7.76, p < 0.001; STS: 0.61%,
t(20) = 59.45, p < 0.001; FDR correction) and scene condition (pF:
0.62%, t(20) = 23.06, p < 0.001; LO: 0.79%, t(20) = 29.20, p < 0.001;
PPA: 0.42%, t(20) = 12.78, p < 0.001; STS: 0.09%, t(20) = 5.70, p <
0.001; FDR correction).

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx 5

Fig. 4. Multivariate classification of scenes using sound-based decoders and the classification performance as a function of the number of voxels resampled. (A) We trained a
pattern classifier to predict the pattern of the scene based on the pattern of sounds. (B) Classification accuracies for different ROI sizes and the optimal result were observed
when the ROI contained all voxels. And one-sample t-test was utilized to test the statistical significance of classification accuracies. The error bars denoted the standard error
of the mean. *p < 0.05, **p < 0.01, ***p < 0.001.

Moreover, we performed the paired t-test statistical analysis on Table 2

The results of within-cue classification in each ROI.
the signal changes in the sound and scene tasks for all four ROIs.
It was found that there were significant differences between the within-cue classification
signal changes of both for each ROI. The signals in the scene con-
object-object scene-scene
dition were observed to be significantly greater than that in the
sound condition in the visual cortex (pF: t(20) = 20.09, p < 0.001; Accuracy SEM t(20) P Accuracy SEM t(20) P
(%) (%)
PPA: t(20) = 13.05, p < 0.001; LO: t(20) = 25.82, p < 0.001), and STS
showed a larger change in the sound task (t(20) = 23.73, p < 0.001) pF 13.082 1.211 0.481 0.636 32.242 1.054 6.871 <0.001
LO 12.823 1.186 0.272 0.788 33.383 1.111 7.544 <0.001
than in the scene task.
PPA 12.910 0.803 0.510 0.616 31.448 1.046 6.163 <0.001
STS 19.320 1.172 5.819 <0.001 28.373 1.030 3.274 0.004
3.3. Multivariate pattern classification analysis One-sample t-tests of classification accuracies for within-cue stimuli in each ROI
(two-tailed). In the 8-way classification of object sounds, the decoding failed in the
The specific activity patterns for the object sounds in each ROI pF, LO and PPA, which distribute in visual cortex. The scenes can be successfully
were calculated to decode the patterns of the scene. Fig. 4A depicted decoded by scenes in all ROIs.
the average of classification accuracies in four ROIs. A one-tailed
t-test analysis for the results suggested that the classification per-
gradually as the size of the voxels increased (except for STS, in
formances were significantly above the chance level (25%) in three
which there was a peak in the number of 500, and the later clas-
regions (pF: 29.68%, t(20) = 6.01, p < 0.001; LO: 29.91%, t(20) = 4.72,
sification results showed a slight decrease with the increase of the
p < 0.001; STS: 27.70%, t(20) = 2.23, p < 0.05; FDR correction). We
voxel numbers, and then it did not increase until voxels in the size
can see that scene-evoked activity patterns could be built based
of 700). However, one-way analysis of variance (ANOVA) showed
on the combination of sound-evoked patterns even if we did not
that there were no significant differences in the classification accu-
tell subjects to pay attention to the objects in the scene before the
racy when the number of the voxels increased in all of the ROIs. The
scan. However, the sound-evoked patterns in PPA cannot construct
classification accuracies in these groups were significantly smaller
the scene-evoked patterns (t(20) = 0.66, p = 0.52). To exclude the
than the group that all voxels were included (p < 0.05).
performance of the uncorrelated brain area on the scene decoding
based on the object sound within the scene, we examined the ret-
rosplenial cortex (RSC) as a control area which has been proved to 3.4. Differences between indoor and outdoor scenes
be involved in scene decoding, however, the literatures have not
suggested that this area may represent the semantic relationship To explore the influence of the scene sub-categories on the scene
between the scene and the sound. The classification accuracy in the recognition, we further divided the scene into indoor and outdoor
control area was 25.92% (t(20) = 1.12, p = 0.27). In addition, the clas- scenes and then re-analyzed the data using the MVPA as described
sifications based on the “shuffled” data failed in all of ROIs (pF: t(20) above. A paired t-test was performed to compare the decoding per-
= -0.64, p = 0.527; PPA: t(20) = -1.19, p = 0.249; LO: t(20) = 0.97, formances of indoor scenes with outdoor scenes in each ROI. The
p = 0.334; STS: t(20) = -0.44, p = 0.665). Furthermore, we per- classification accuracies for the indoor scenes were found to be sig-
formed paired t-test between classification accuracies of “true” data nificantly lower than that in the outdoor scenes in pF (t(20) = 2.52,
and “shuffle” data: pF: t(20) = -5.58, p < 0.01; PPA: t(20) = -5.98, p < 0.05), PPA (t(20) = 4.04, p < 0.01) and STS (t(20) = 3.93, p < 0.01).
p < 0.001; LO: t(20) = 0.92, p = 0.632; STS: t(20) = -3.12, p = 0.04. However, there was no significant difference in LO (t(20) = 1.03, p =
Therefore, statistical tests verified that the successful decoding was 0.313). Fig. 5 showed the details of the statistical analysis.
reliable in pF, LO and STS.
The results of the within-cue classification analysis were shown 3.5. FC analyses in the sound and scene tasks
in Table 2. The 8-way classification of sounds only succeeded in
STS (t(20) = 5.819, p < 0.001), while the classification accuracies of To investigate the integration differences among four ROIs in
scene-to-scene were significantly higher than the chance-level the sound and scene tasks, we performed a ROI-to-ROI FC analysis
(25%). Furthermore, to diminish the influence of the different ROI (Fig. 6).
sizes, we used F score to select different numbers of voxels in 4 ROIs In the scene task, no significant FC was observed between STS
(from 50 to 950). From Fig. 4B, the classification accuracy increased and PPA (STS vs PPA: t(20) = -1.76, p = 0.093). On the contrary,

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
6 X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx
Fig. 5. Classification results for indoor and outdoor scenes. The scenes were divided
into indoor and outdoor scenes (indoor: kitchen and office; outdoor: grass and
street), and the classification process was repeated. The paired t-test was performed
between indoor and outdoor scenes. The error bars denoted the standard error of
the mean. *p < 0.05, **p < 0.01, ***p < 0.001.
Fig. 6. ROI-to-ROI FC analyses among 4 ROIs (pF, LO, PPA and STS) in the sound
and scene tasks. In order to show the connectivity more clearly, the ROIs are shown
as points in different locations, even though both hemispheres areas were used in
the FC analysis. The line between two regions stands for a significant FC between
regions. “Blue” represented the negative correlation; “Red” expressed the positive
correlation. L, left; R, right (For interpretation of the references to colour in this
figure legend, the reader is referred to the web version of this article).
significant FC were found between pF and LO, PPA, STS (pF vs LO:
t(20) = 20.62, p < 0.001; pF vs PPA: t(20) = 10.00, p < 0.001; pF vs STS:
t(20) = -5.35, p < 0.001). And there were significant FC between LO
and PPA, STS (LO vs PPA: t(20) = 7.87, p < 0.001; LO vs STS: t(20) =
-3.92, p < 0.001).
In the sound task, we did not find significant FC between STS and
pF, LO, PPA (STS vs pF: t(20) = 0.99, p = 0.33; STS vs LO: t(20) = -0.49,
p = 0.63; STS vs PPA: t(20) = -1.66, p = 0.11). In contrast, significant
FC was observed between pF and LO, PPA (pF vs LO: t(20) = 14.06,
p < 0.001; pF vs PPA: t(20) = 7.74, p < 0.001). And the FC was also
significant between LO and PPA (LO vs PPA: t(20) = 5.75, p < 0.001).
The above ROI-to-ROI FC analyses found that STS showed no
significant positive correlation with the other three regions both
in the sound and scene tasks. To explore the functional role of STS
in the scene and sound task, we made a further seed-to-voxel FC
analysis using STS as the seed. More specific regions were found
Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
to showed a significant FC with STS, including speech aspects areas
(insular cortex), visuospatial imagery and memory relevant regions
(middle frontal gyrus), semantic aspects areas (posterior middle
temporal gyrus, inferior frontal gyrus), objects sensitive areas (LOC)
and audio-visual areas (posterior superior temporal gyrus). Fig. 7
illustrated the brain regions that were significantly correlated with
STS in both tasks (p < 0.001, FDR correction), and details of the brain
regions were shown in Table 3.
3.6. Searchlight analysis
To examine the areas that could show above-chance decoding
accuracy outside our pre - defined ROIs, we performed a search-
light analysis to identify regions containing significant associations
between scene patterns and patterns evoked by object sounds. We
defined a 5-mm - radius spherical mask centered on each voxel
in the brain and conducted the scene - from - sounds classification
procedures as described in MVPA classification. Fig. 8 showed the
results of the pattern classification analysis. Compared with the
seed-to-voxel FC analysis, more voxels were found in the frontal
cortex, even though they were scattered. Other than these areas,
the voxels with significant classification accuracies for scenes were
mainly distributed in precentral gyrus, as found in the FC analysis.
Considering that the significant classification accuracies for scenes
in these areas, it is worth to explore the connection between them
during the sound-to-scene presentation, which will be our future
work.
4. Discussion
The principal finding of this study is that the scene patterns can
be decoded based on the relevant sound-evoked information in pF,
LO and STS. We used the activity patterns of the two object sounds
strongly associated with the scene to predict the scene patterns
successfully. However, the similar relationship between the scene
and sound was not observed in PPA. And a further FC analysis using
the four ROIs as seeds indicated that there was no significant FC
between STS and the other three regions in the sound and scene
tasks. Through exploring the construction of the scene patterns
from the response patterns of the sound in pF, LO, and STS, our
findings could provide evidence for the neural mechanism in scene
recognition based on the sound-evoked information.
4.1. Reconstructing scenes from object sounds using MVPA
In this study, we found that the patterns evoked by object sounds
could discriminate the corresponding patterns of the scene signif-
icantly in pF, LO and STS, but not in the PPA.
The ventral occipital-temporal cortex (VOTC) has been explored
both in sighted and congenitally blind individuals, and agreed with
the notion that this region could represent objects in a multi-modal
way (Bi et al., 2016). By investigating the relationship between
the actual visual stimuli and the fMRI activity in early visual areas
based on quantitative receptive-field models, one previous study
suggested that it was possible to construct the visual stimuli from
the fMRI activity patterns (Kay et al., 2008). Another study recon-
structed the complex natural scenes successfully by developing a
new Bayesian decoder on the basis of the fMRI signals in the early
and anterior visual areas (Naselaris et al., 2009). And the existing
literature found that the patterns of the scene could be predicted
successfully from the patterns of the within-scene objects in LO
(MacEvoy and Epstein, 2011). However, our study showed that the
patterns evoked by the scene could be decoded by the patterns of
the scene-relevant sounds using MVPA.
One recent study (Klemen and Chambers, 2012) suggested that
LOC could decode the object sounds because the sounds evoked
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx 7

Fig. 7. Surface displays for seed-to-voxel FC analyses using STS as a seed in the scene and sound tasks separately. Threshold: voxel-level p = 0.001, FWE correction.

Table 3
Regions that showed significant functional connectivity with STS in the scene and sound tasks.

Regions of Numbers of MNI coordinates peak

Seed intensity
FC voxels
x y z t-value
Scene conditions: Positive correlations
correlations
STS R STG: PreCG, PostCG, ACC, IC R, CO R, MTG R, 6089 63 −9 3 16.76
PCu, PO R, PT R, SMA, pSTG R, pSMG R, aSMG R,
SPL, HG R, iLOC R, Amy, aSTG R, SFG, pMTG
L STG: IC L, PreCG L, PostCG L, MTG L, SMG L, 3899 −48 −12 0 26.40
IPL, pSTG L, iLOC L, HG L, Tha, Amy L
L Cereb8 137 −15 −60 −51 7.67
L MFG 106 −36 42 30 6.48
R IFG 35 42 39 6 4.45
R Cereb8 34 15 −72 −54 8.03
Sound Conditions: Positive correlations
STS R STG: PreCG, PostCG, IC, CO, MFG, MTG, PO, 11884 60 −24 9 25.06
PT, IFG, SMA, ACC, pSTG, aSMG, pSMG, HG,
aSMG, SFG, IFG, iLOC L, PaCiG, aSTG, Amy L,
pMTG R, FOC R
L Cereb8: Cereb6 L, Cereb2 L, Cereb1 L, Cereb7 L 648 −24 −63 −51 9.28
R Cereb8: Cereb6 R 336 15 −75 −51 7.71
L SPL 82 −30 −57 66 5.47

The MNI coordinates and selected the clusters (≥ 30 voxels) that included t-values of peak voxel representing the significant FC with the STS. The clusters homologous to the
peak voxel are written in bold and the other areas contained in this cluster are written in regular. PreCG: precentral gyrus, PostCG: postcentral gyrus, ACC: anterior cingulate
cortex, IC: insular Cortex, CO: central opercular cortex, M/STG: middle/superior Temporal gyrus, PCu: precuneus, PO: parietal operculum cortex, PT: planum temporale, SMA:
supplementary motor cortex, SMG: supramarginal gyrus, PL: parietal lobule, HG: heschl’s gyrus, LOC: lateral occipital cortex, Amy: amygdala, M/SFG: middle/superior frontal
gyrus, I/SPL: inferior/superior parietal lobule, MTG: middle temporal gyrus, SMG: supramarginal Gyrus, pSTG: posterior superior temporal gyrus, HG: heschl’s gyrus, Tha:
thalamus, S/M/IFG: superior/middle/inferior frontal gyrus, PaCiG: paracingulate gyrus, FOC: orbital frontal cortex, Cereb: cerebellum. a/p/i: anterior/posterior/inferior; L/R:
left/right.

subjects’ imaging of the corresponding objects, and they used the
fMRI data of imaginary objects to determine the identity of the
objects successfully, which provided consistent evidence that the
imagery and actual perception may share the same neural mecha-
nism (Kosslyn et al., 2001; Hubbard, 2010), which even was verified
by the successful scene decoding during perception and imagery
(Johnson and Johnson, 2014). One previous study found that two
subregions within the LOC, pF and LO were involved in processing
different aspects of visual recognition (Nordhjem et al., 2015). In
our study, we speculated that one explanation for the successful
scene decoding based on the sounds may be that the sound stim-
uli could induce our imaging the corresponding objects (Klemen
and Chambers, 2012; Vetter et al., 2014), and there was a strong
association between the objects and sounds within the scenes.
As opposed to LOC, we did not find the semantic relationship
between the sound and the scene in PPA. That is to say, even if many
studies have attested the important role of PPA in scene recognition,
reactions to the scene in this area seemed to have nothing to do with
the response to the corresponding sound. Other studies demon-
strated that PPA could also represent the scene-relevant object
information (Macevoy and Epstein, 2009; Linsley and MacEvoy,
2014) and take part in tuning the object information (Macevoy and
Epstein, 2009; Harel et al., 2013). Although subjects might imagine
the objects and animals when hearing sounds, activity patterns in
PPA only included the information of the standalone objects and
this information disappeared when the objects arose in the scene
(MacEvoy and Epstein, 2011).
A growing number of studies hold the notion that integrating the
stimuli from the multisensory modalities was the inherent nature

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
8 X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx

Fig. 8. Searchlight-based classification analysis maps across subjects. Surface displays the decoding accuracy of voxels with significant classification accuracies higher than
chance-level (25%, P < 0.01, FDR correction). (A) Outlined regions are STS (dark blue), pF (light blue), PPA (purple) and the LO (green). (B) Besides of the four ROIs, some main
regions are labeled, including precentral gyrus, MFG, and IFG (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of
this article).

of the human brain (Ghazanfar and Schroeder, 2006; van Atteveldt
et al., 2014), and multisensory processing may happen from the
initial narrow mechanism to highly flexible various levels (Muckli
et al., 2015). STS played a central role in multisensory interaction
and responded to the visual and auditory objects strongly (Klemen
and Chambers, 2012). Our study showed that STS could also repre-
sent the semantic relationship between the scene and sound, and
we speculated the possibility that one could imagine the animals
and man-made manipulable objects (tools) contained in the scenes
when hearing the sound stimuli, for example, “moo” and Cattle in
the grass, “vroom” and automobile engine in the street. Further-
more, the STS has been believed to be capable of integrating the
meaningful information of audio-visual objects into a coherent per-
cept (Beauchamp et al., 2004; Venezia et al., 2017), and take part in
processing the many diverse aspects, for instance, theory of mind,
audiovisual integration, motion processing, speech processing and
face processing (Hein and Knight, 2008; Blank and von Kriegstein,
2013). And the study considered that the same brain region might
be responsible for different recognition functions under the distinct
task network (Hein and Knight, 2008; Watson et al., 2014). There-
fore, STS may also be involved in tuning the sound information in
the scene.
Moreover, our study found that there were significant differ-
ences in the signal change between the scene and scene-related
sounds in all four regions. Specifically, the signal change of the scene
images was significantly greater than that of the sounds in pF, LO
and PPA, while the signal change of the sounds was significantly
higher than that of the scene signal in STS. In spite of the significant
differences in the signal change, diff ;erent scenes could be suc-
cessfully discriminated based on the patterns of sounds of objects
within the scenes in ROIs, which proved that these brain regions
could represent the sound-scene semantic relationship. One expla-
nation for the results may be that the univariate analysis of fMRI
data were not as sensitive as MVPA (Norman et al., 2006; Stelzer
et al., 2013) to the small number of neurons that shared a little
classification information.
4.2. Influence of scene openness
Previous studies suggested that the scene openness might influ-
ence the scene recognition by comparing the fMRI activities of the
indoor and outdoor scenes, and the neural signal induced by indoor
scenes was found to be stronger than that by outdoor scenes in the
PPA (Henderson et al., 2007; Kravitz et al., 2011). In this study, we
further analyzed the indoor and outdoor scenes using MVPA and
found that the openness of the scenes could affect the sound-based
scene prediction in pF, PPA and STS, which was consistent with the
previous studies. However, the LO showed a great tolerance of the
scene openness in the present study. In addition, the classification
analysis of indoor and outdoor scenes suggested that pF, PPA and
STS might be able to represent the semantic relationship between
the outdoor scenes and associated sounds, but failed to process the
association between indoor scenes and the sounds.
4.3. FC in the scene and sound tasks
Using the ROI-to-ROI FC analyses, we found that there were
significant functional correlations between these four ROIs, which
formed two distinct sub-networks (negative correlation network
consisted of pF, LO and STS; positive correlation network com-
posed of pF, LO and PPA) in the scene task. PPA was showed to
be more sensitive to the scenes than other visual stimuli (Epstein
et al., 1999, 2003; Epstein et al., 2006), and involved in decoding the
appearance and layout of the scenes (Epstein et al., 1999). Previous
studies (Macevoy and Epstein, 2009; MacEvoy and Epstein, 2011)
found that the brain regions could classify the four categories of the
scenes successfully containing the pF and LO, which indicated that
these two regions were also engaged in scene classification. The
aforementioned evidence was consistent with our FC analyses dur-
ing the scene processing. But an opposite situation was observed in
the sound task, in which pF, LO and PPA formed one sub-network,
while STS showed no functional associations with the other three
regions. The STS was suggested to represent stimuli from visual,
auditory, and audio-visual modality (Beauchamp, 2005; Venezia
et al., 2017). There existed a question, namely why STS was not

Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx
functionally connected with the other three regions in the sound
task. We speculated the reason was that only STS participated in
representing the sound stimuli, therefore, it was independent of the
network formed by the other three ROIs which mainly represented
the visual scenes.
As STS showed an independent role in the ROI-to-ROI analy-
ses, we further explored brain regions that were related to STS in
the sound and the scene tasks. Some common brain areas that were
functionally cooperated with STS in both tasks suggested that these
regions might be involved in different functions in different tasks
(Beauchamp et al., 2004; Hein and Knight, 2008; Venezia et al.,
2017). In this study, the insular cortex showed a significant FC with
STS. The insular cortex has been demonstrated to play a role in the
cross-modal coincidence and matching (Calvert, 2001; Senkowski
et al., 2007), and was involved in a number of different functions
such as pain perception, speech production and social emotion pro-
cessing (Nieuwenhuys, 2011; Gu et al., 2013). The part of the middle
frontal gyrus was proved to store and process the working mem-
ory (Leung et al., 2002; Senkowski et al., 2007). One recent study
(Whitney et al., 2010) indicated that posterior middle temporal
gyrus (pMTG) and inferior frontal gyrus (IFG) played an impor-
tant role in the semantic control by using repetitive transcranial
magnetic stimulation (rTMS) to obstruct the processing of IFG and
pMTG. Additional evidence (MacEvoy and Epstein, 2011) pointed
that LOC could represent the semantic relationship between the
scenes and scene-relevant objects and provided an object-based
channel to decode the scene. Previous studies showed that the pos-
terior superior temporal gyrus could integrate the different types
of within-modality and cross-modality information (Beauchamp
et al., 2004; Venezia et al., 2017). It is possible that the key assign-
ment in the experiment offered some category information to these
brain areas, even though the data during the motor response were
not used for decoding. Considering that part of motor responses
can overlap in time with the responses induced by sound, the key
assignment can be improved by setting different motor responses
to a same scene in future study, which may reduce the influ-
ence of fixed motor responses on stimuli classification. In addition,
the existence of the common brain regions may imply that these
regions cooperated with STS during processing the sound and the
scene to represent the semantic relationship between both. And the
connectivity between these areas can help us understand the infor-
mation flowing during the sound-to-scene representation, which
will be our next work.
5. Conclusions
In this study, we explored the scene decoding based on the pat-
terns of sounds of objects within the scenes and found that pF,
LO and STS could represent the semantic relationship between the
scenes and the sounds of the associated objects. However, this
semantic association was not observed in PPA. Furthermore, by
dividing the scenes into indoor and outdoor parts, we found that
LO was not sensitive to the openness of the scenes. The ROI-to-ROI
FC analyses among four ROIs in the scene and sound tasks indi-
cated that STS did not coordinate with the other three ROIs in both
tasks, and a further seed-to-voxel analysis using STS as the seed sug-
gested a distinct network in processing the scenes and sounds. In
summary, our study showed the existence of a cross-modal sound-
based channel for scene decoding in the occipitotemporal cortex,
which could give a further access to understand the neural mech-
anism of scene recognition.
Conflict of interest
No conflict of interest.
Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
9
Acknowledgments
This work was supported by the National Natural Science
Foundation of China (No. U1736219, No. 61860206010 and
No.61571327), Shandong Provincial Natural Science Foundation
of China (No. ZR2015HM081) and Project of Shandong Province
Higher Educational Science and Technology Program (J15LL01).
References
Amedi, A., von Kriegstein, K., van Atteveldt, N.M., Beauchamp, M., Naumer, M.J., 2005.
Functional imaging of human crossmodal identification and object recognition.
Exp. Brain Res. 166, 559–571.
Beauchamp, M.S., 2005. See me, hear me, touch me: multisensory integration in
lateral occipital-temporal cortex. Curr. Opin. Neurobiol. 15, 145–153.
Beauchamp, M.S., Lee, K.E., Argall, B.D., Martin, A., 2004. Integration of auditory
and visual information about objects in superior temporal sulcus. Neuron 41,
809–823.
Belin, P., Zatorre, R.J., Lafaille, P., Ahad, P., Pike, B., 2000. Voice-selective areas in
human auditory cortex. Nature 403, 309–312.
Bi, Y., Wang, X., Caramazza, A., 2016. Object domain and modality in the ventral
visual pathway. Trends Cogn. Sci. (Regul. Ed.) 20, 282–290.
Blank, H., von Kriegstein, K., 2013. Mechanisms of enhancing visual-speech recog-
nition by prior auditory information. NeuroImage 65, 109–118.
Calvert, G.A., 2001. Crossmodal processing in the human brain: insights from func-
tional neuroimaging studies. Cereb. Cortex 11, 1110–1123.
Chen, Y.W., Lin, C.J., 2006. Combining SVMs with various feature selection strategies.
Studies in Fuzziness & Soft Computing 207, 315–324.
Doehrmann, O., Weigelt, S., Altmann, C.F., Kaiser, J., Naumer, M.J., 2010. Audiovi-
sual functional magnetic resonance imaging adaptation reveals multisensory
integration effects in object-related sensory cortices. J. Neurosci. 30, 3370–3379.
Drucker, D.M., Aguirre, G.K., 2009. Different spatial scales of shape similarity repre-
sentation in lateral and ventral LOC. Cereb. Cortex 19, 2269–2280.
Epstein, R., Kanwisher, N., 1998. A cortical representation of the local visual envi-
ronment. Nature 392, 598–601.
Epstein, R., Harris, A., Stanley, D., Kanwisher, N., 1999. The parahippocampal place
area: Recognition, navigation, or encoding? Neuron 23, 115–125.
Epstein, R., Graham, K.S., Downing, P.E., 2003. Viewpoint-specific scene representa-
tions in human parahippocampal cortex. Neuron 37, 865–876.
Epstein, R.A., Higgins, J.S., Parker, W., Aguirre, G.K., Cooperman, S., 2006. Cortical
correlates of face and scene inversion: a comparison. Neuropsychologia 44,
1145–1158.
Epstein, R.A., Parker, W.E., Feiler, A.M., 2007. Where am I now? Distinct roles for
parahippocampal and retrosplenial cortices in place recognition. J. Neurosci. 27,
6141–6149.
Ghazanfar, A.A., Schroeder, C.E., 2006. Is neocortex essentially multisensory? Trends
Cogn. Sci. (Regul. Ed.) 10, 278–285.
Greene, M.R., Oliva, A., 2009. Recognition of natural scenes from global properties:
seeing the forest without representing the trees. Cogn. Psychol. 58, 137–176.
Gu, X., Hof, P.R., Friston, K.J., Fan, J., 2013. Anterior insular cortex and emotional
awareness. J. Comp. Neurol. 521, 3371–3388.
Harel, A., Kravitz, D.J., Baker, C.I., 2013. Deconstructing visual scenes in cortex: gra-
dients of object and spatial layout information. Cereb. Cortex 23, 947–957.
Harrison, S.A., Tong, F., 2009. Decoding reveals the contents of visual working mem-
ory in early visual areas. Nature 458, 632–635.
Haushofer, J., Livingstone, M.S., Kanwisher, N., 2008. Multivariate patterns in object-
selective cortex dissociate perceptual and physical shape similarity. PLoS Biol.
6, e187.
Haxby, J.V., Gobbini, M.I., Furey, M.L., Ishai, A., Schouten, J.L., Pietrini, P., 2001.
Distributed and overlapping representations of faces and objects in ventral tem-
poral cortex. Science 293, 2425–2430.
Hein, G., Knight, R.T., 2008. Superior temporal sulcus—it’s my area: or is it? Cognitive
Neuroscience. Journal of 20, 2125–2136.
Henderson, J.M., Larson, C.L., Zhu, D.C., 2007. Cortical activation to indoor versus
outdoor scenes: an fMRI study. Exp. Brain Res. 179, 75–84.
Hubbard, T.L., 2010. Auditory imagery: empirical findings. Psychol. Bull. 136,
302–329.
Johnson, M.R., Johnson, M.K., 2014. Decoding individual natural scene representa-
tions during perception and imagery. Front. Hum. Neurosci. 8, 59.
Kay, K.N., Thomas, N., Prenger, R.J., Gallant, J.L., 2008. Identifying natural images
from human brain activity. Nature 452, 352–355.
Klemen, J., Chambers, C.D., 2012. Current perspectives and methods in studying
neural mechanisms of multisensory interactions. Neurosci. Biobehav. Rev. 36,
111–133.
Kosslyn, S.M., Ganis, G., Thompson, W.L., 2001. Neural foundations of imagery. Nat.
Rev. Neurosci. 2, 635–642.
Kravitz, D.J., Peng, C.S., Baker, C.I., 2011. Real-world scene representations in
high-level visual cortex: it’s the spaces more than the places. J. Neurosci. 31,
7322–7333.
Lalli, S., Piacentini, S., Franzini, A., Panzacchi, A., Cerami, C., Messina, G., Ferré, F.,
Perani, D., Albanese, A., 2012. Epidural premotor cortical stimulation in pri-
mary focal dystonia: clinical and 18F-fluoro deoxyglucose positron emission
tomography open study. Mov. Disord. 27, 533–538.
G Model
NSR-4243; No. of Pages 10 ARTICLE IN PRESS
10 X. Wang et al. / Neuroscience Research xxx (2018) xxx–xxx
Leung, H., Gore, J.C., Goldman-Rakic, P.S., 2002. Sustained mnemonic response in
the human middle frontal gyrus during on-line storage of spatial memoranda.
Cognitive Neuroscience, Journal of 14, 659–671.
Liang, M., Mouraux, A., Hu, L., Iannetti, G.D., 2013. Primary sensory cortices contain
distinguishable spatial patterns of activity for each sense. Nat. Commun. 4, 1979.
Linsley, D., MacEvoy, S.P., 2014. Evidence for participation by object-selective visual
cortex in scene category judgments. J. Vis. 14, 19-19.
Macevoy, S.P., Epstein, R.A., 2009. Decoding the representation of multiple simul-
taneous objects in human occipitotemporal cortex. Current Biology Cb 19,
943–947.
MacEvoy, S.P., Epstein, R.A., 2011. Constructing scenes from objects in human occip-
itotemporal cortex. Nat. Neurosci. 14, 1323–1329.
Malach, R., Reppas, J.B., Benson, R.R., Kwong, K.K., Jiang, H., Kennedy, W.A., Ledden,
P.J., Brady, T.J., Rosen, B.R., Tootell, R.B., 1995. Object-related activity revealed by
functional magnetic resonance imaging in human occipital cortex. Proceedings
of the National Academy of Sciences 92, 8135–8139.
Mesulam, M.M., 1998. From sensation to cognition. Brain 121 (Pt 6), 1013–1052.
Muckli, L., Vizioli, L., Petro, L., De Martino, F., Vetter, P., 2015. Predictive coding of
auditory and contextual information in early visual cortex-evidence from layer
specific fMRI brain reading. J. Vis. 15, 720-720.
Naselaris, T., Prenger, R.J., Kay, K.N., Oliver, M., Gallant, J.L., 2009. Bayesian recon-
struction of natural images from human brain activity. Neuron. Neuron 63,
902–915.
Nieuwenhuys, R., 2011. The insular cortex: a review. Prog. Brain Res. 195, 123–163.
Nordhjem, B., Curcic-Blake, B., Meppelink, A.M., Renken, R.J., de Jong, B.M., Leenders,
K.L., van Laar, T., Cornelissen, F.W., 2015. Lateral and medial ventral occipitotem-
poral regions interact during the recognition of images revealed from noise.
Front. Hum. Neurosci. 9, 678.
Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V., 2006. Beyond mind-reading: multi-
voxel pattern analysis of fMRI data. Trends Cogn. Sci. (Regul. Ed.) 10, 424–430.
Park, S., Brady, T.F., Greene, M.R., Oliva, A., 2011. Disentangling scene content from
spatial boundary: complementary roles for the parahippocampal place area
and lateral occipital complex in representing real-world scenes. J. Neurosci. 31,
1333–1340.
Park, S., Konkle, T., Oliva, A., 2015. Parametric coding of the size and clutter of natural
scenes in the human brain. Cereb. Cortex 25, 1792–1805.
Renninger, L.W., Malik, J., 2004. When is scene identification just texture recogni-
tion? Vision Res. 44, 2301–2311.
Rohe, T., Noppeney, U., 2016. Distinct computational principles govern multisensory
integration in primary sensory and association cortices. Curr. Biol. 26, 509–514.
Please cite this article in press as: Wang, X., et al., Decoding natural scenes based on sounds of objects within scenes using multivariate
pattern analysis. Neurosci. Res. (2018), https://doi.org/10.1016/j.neures.2018.11.009
View publication stats
Said, C.P., Moore, C.D., Engell, A.D., Todorov, A., Haxby, J.V., 2010. Distributed repre-
sentations of dynamic facial expressions in the superior temporal sulcus. J. Vis.
10, 11-11.
Senkowski, D., Saint-Amour, D., Kelly, S.P., Foxe, J.J., 2007. Multisensory processing
of naturalistic objects in motion: a high-density electrical mapping and source
estimation study. NeuroImage 36, 877–888.
Stansbury, D.E., Naselaris, T., Gallant, J.L., 2013. Natural scene statistics account
for the representation of scene categories in human visual cortex. Neuron 79,
1025–1034.
Stelzer, J., Chen, Y., Turner, R., 2013. Statistical inference and multiple testing cor-
rection in classification-based multi-voxel pattern analysis (MVPA): random
permutations and cluster size control. NeuroImage 65, 69–82.
Tyll, S., Bonath, B., Schoenfeld, M.A., Heinze, H.J., Ohl, F.W., Noesselt, T., 2013. Neural
basis of multisensory looming signals. NeuroImage 65, 13–22.
van Atteveldt, N., Murray, M.M., Thut, G., Schroeder, C.E., 2014. Multisensory inte-
gration: flexible use of general operations. Neuron 81, 1240–1253.
Venezia, J.H., Vaden Jr., K.I., Rong, F., Maddox, D., Saberi, K., Hickok, G., 2017. Auditory,
visual and audiovisual speech processing streams in superior temporal sulcus.
Front. Hum. Neurosci. 11, 174.
Vetter, P., Smith, F.W., Muckli, L., 2014. Decoding sound and imagery content in early
visual cortex. Curr. Biol. 24, 1256–1262.
Walther, D.B., Caddigan, E., Fei-Fei, L., Beck, D.M., 2009. Natural scene categories
revealed in distributed patterns of activity in the human brain. J. Neurosci. 29,
10573–10581.
Walther, D.B., Chai, B., Caddigan, E., Beck, D.M., Fei-Fei, L., 2011. Simple line drawings
suffice for functional MRI decoding of natural scene categories. Proc Natl Acad
Sci U S A 108, 9661–9666.
Watson, R., Latinus, M., Charest, I., Crabbe, F., Belin, P., 2014. People-selectivity,
audiovisual integration and heteromodality in the superior temporal sulcus.
Cortex 50, 125–136.
Werner, S., Noppeney, U., 2010. Distinct functional contributions of primary sen-
sory and association areas to audiovisual integration in object categorization. J.
Neurosci. 30, 2662–2675.
Whitfield-Gabrieli, S., Nieto-Castanon, A., 2012. Conn: a functional connectivity tool-
box for correlated and anticorrelated brain networks. Brain connect 2: 125-141.
Brain Connect. 2, 125–141.
Whitney, C., Kirk, M., O’Sullivan, J., Ralph, M.A.L., Jefferies, E., 2010. The neural orga-
nization of semantic control: TMS evidence for a distributed network in left
inferior frontal and posterior middle temporal gyrus. Cereb. Cortex, bhq180.