IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL.
60, 2022
Hyperspectral and LiDAR Data Classification Using
Joint CNNs and Morphological Feature Learning
Swalpa Kumar Roy , Student Member, IEEE, Ankur Deria , Danfeng Hong , Senior Member, IEEE,
Muhammad Ahmad , Antonio Plaza , Fellow, IEEE, and Jocelyn Chanussot , Fellow, IEEE
Abstract— Convolutional neural networks (CNNs) have been
extensively utilized for hyperspectral image (HSI) and light
detection and ranging (LiDAR) data classification. However,
CNNs have not been much explored for joint HSI and LiDAR
image classification. Therefore, this article proposes a joint
feature learning (HSI and LiDAR) and fusion mechanism using
CNN and spatial morphological blocks, which generates highly
accurate land-cover maps. The CNN model comprises three
Conv3D layers and is directly applied to the HSIs for extracting
discriminative spectral–spatial feature representation. On the
contrary, the spatial morphological block is able to capture
the information relevant to the height or shape of the different
land-cover regions from LiDAR data. The LiDAR features are
extracted using morphological dilation and erosion layers that
increase the robustness of the proposed model by considering
elevation information as an additional feature. Finally, both
the obtained features from CNNs and spatial morphological
blocks are combined using an additive operation prior to the
classification. Extensive experiments are shown with widely used
HSIs and LiDAR datasets, i.e., University of Houston (UH),
Trento, and MUUFL Gulfport scene. The reported results show
that the proposed model significantly outperforms traditional
methods and other state-of-the-art deep learning models. The
source code for the proposed model will be made available
publicly at https://github.com/AnkurDeria/HSI+LiDAR.
Index Terms— Convolutional neural networks (CNNs), hyper-
spectral image (HSI) classification, light detection and ranging
(LiDAR).
I. I NTRODUCTION
H YPERSPECTRAL images (HSIs) convey rich spectral
information, which is encoded in hundreds of narrow and
Manuscript received December 9, 2021; revised March 12, 2022; accepted
May 21, 2022. Date of publication May 23, 2022; date of current version
June 10, 2022. This work was supported in part by the National Natural
Science Foundation of China under Grant 62161160336 and Grant 42030111;
in part by the MIAI@Grenoble Alpes under Grant ANR-19-P3IA-0003; and
in part by the AXA Research Fund, by the Spanish Ministerio de Ciencia e
Innovación under Project PID2019-110315RB-I00 (APRISA). (Corresponding
author: Danfeng Hong.)
Swalpa Kumar Roy and Ankur Deria are with the Department of
Computer Science and Engineering, Jalpaiguri Government Engineer-
ing College, Jalpaiguri 735102, India (e-mail: swalpa@cse.jgec.ac.in;
ad2207@cse.jgec.ac.in).
Danfeng Hong is with the Key Laboratory of Computational Optical Imag-
ing Technology, Aerospace Information Research Institute, Chinese Academy
of Sciences, Beijing 100094, China (e-mail: hongdf@aircas.ac.cn).
Muhammad Ahmad is with the Department of Computer Science,
National University of Computer and Emerging Sciences, Islamabad,
Chiniot-Faisalabad Campus, Chiniot 35400, Pakistan (e-mail: mahmad00@
gmail.com).
Antonio Plaza is with the Hyperspectral Computing Laboratory, Department
of Technology of Computers and Communications, Escuela Politécnica,
University of Extremadura, 10003 Cáceres, Spain (e-mail: aplaza@unex.es).
Jocelyn Chanussot is with the GIPSA-Lab, CNRS, Grenoble INP, Université
Grenoble Alpes, 38000 Grenoble, France (e-mail: jocelyn@hi.is).
Digital Object Identifier 10.1109/TGRS.2022.3177633
1558-0644 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
5530416
contiguous spectral bands [1], [2]. The robust discriminative
behavior of the spectral features helps to capture subtle spec-
tral differences, and thus, the classification task with HSIs has
become one of the key aspects in several applications related to
Earth observation, such as deforestation, land-cover mapping,
and mineral exploration (to mention a few) [3]–[5]. However,
HSIs are unable to properly differentiate between two complex
land covers as long as they produce similar spectral responses.
To illustrate this point, we plot some spectral signatures
from the well-known University of Houston HSI dataset.
It can be observed that classes such as Healthy Grass have
spectral signatures that are similar to those of Stressed Grass,
as shown in Fig. 1. Other classes such as Highway, Parking
Lot 1, Railway, Residential, and Road also exhibit very similar
spectral signatures. This poses a challenge in the classification
of HSIs, as the classifiers struggle to distinguish similar land
covers. There often exist such types of complex land-cover
types in urban and rural areas, and this problem can be easily
overcome by using another source of information, obtained
from light detection and ranging (LiDAR) data.
LiDAR data can provide elevation information of height and
shape of the image surface with respect to the sensor [6], thus
providing complementary information for HSI data, and can,
therefore, be useful to distinguish the land covers that are made
up of the same material. Using LiDAR, one can accurately
classify between roofs and roads even when they have similar
materials. However, LiDAR data fail to distinguish between
the classes in which the objects contain similar elevation
information, for example, two roofs having similar elevation
information with different materials, i.e., concrete or asphalt.
Developing HSI and LiDAR data fusion techniques represents
an interesting and challenging research topic to explore, whose
performance has already been studied in the literature for
land-cover classification [6]–[8].
In the early days, traditional methods were widely used
for HSI classification, which showed significant performance
gains even with limited training [9]–[11]. These methods
extract and learn the feature representation and then fit those
features into a machine learning model for classification [1],
[8], [12]–[14]. Among the traditional models, the support
vector machine (SVM) with a nonlinear kernel is widely used
one, especially when the training samples are limited [15],
[16]. To further improve the performance, some handcrafted
features are designed before the use of the classifier for
unlabeled HSI data [17]. However, the performance achieved
was not satisfactory due to the nonlinear relation between the
captured spectral information and that of the corresponding
5530416
Fig. 1. Classwise spectral signatures of the classes such as Healthy grass, Stressed grass, Highway, Residential, and Road extracted from the University of
Houston (UH) hyperspectral dataset. (a) Healthy grass. (b) Stressed grass. (c) Highway. (d) Parking Lot 1. (e) Railway. (f) Residential. (g) Road.
object of HSI data, which makes accurate classification more
challenging for traditional methods. Zhang et al. introduced
polygonal partition to capture geometrical shape features of
land covers in real-life environments and adopted spectral
self-similarity criteria for enlarging the number of training
samples, which led to better classification performance in
limited training scenarios [18]. Duan et al. [19] extracted
discriminative feature structural profiles from the raw HSIs
and utilized a decision label fusion strategy for accurate clas-
sification. Kang et al. [20] introduced redundancy-free and
discriminative HSI features based on the intrinsic image
decomposition strategy for improved classification.
Recently, deep learning models have shown great suc-
cess in land-use and land-cover (LULC) classification tasks
and outperformed traditional methods due to their automatic
capability to achieve high- and low-level feature learning
using its trainable kernels. For instance, convolutional neural
networks (CNNs) extract feature maps that are invariant to
the local changes with respect to their input [21]. A CNN
framework that extracts the contextual feature representa-
tion to capture more discriminative spectral information and
spatial information was introduced by Lee and Kwon [22].
Hong et al. considered the spatial relationship between pixels
by measuring spectral similarities and proposed a miniGCN
for HSI classification, achieving the state-of-the-art classifica-
tion accuracies [23], [24].
Due to their larger depth, most of the deep networks
suffer from the vanishing gradient problem, and to tackle this,
a residual network (ResNet) is introduced, which can ensure
minimum information loss after each convolutional opera-
tion [25]. Zhong et al. [26] proposed a 3-D spectral–spatial
residual network (SSRN) for better modeling spectral and
spatial features. Extended models are proposed based on light-
weight spectral–spatial squeeze-and-excitation attention [27],
adaptive kernels [28] and pyramidal residual networks [29],
lightweight heterogeneous kernel convolution [30], and gradi-
ent centralized convolution [31], [32], which enables efficient
feature extraction and classification. To improve the former
models, Roy et al. proposed a sequential model to establish
a relation between the extracted feature from 3-D and 2-D
convolutions [33]. Recently, Ahmad et al. explored the atten-
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
tion mechanism with a hybrid dense network with different
kernels in each block to enhance the feature learning for HSI
classification [34]. In order to capture the shape information
of arbitrary land-cover texture classes, morphological dilation
and erosion layers with trainable kernels are employed with
its trainable kernels for accurate classification of HSIs [35].
Though the HSIs exhibit rich spectral–spatial information,
the lack of elevation information led to poor classification
performance in a complex scene. To overcome this issue,
LiDAR data can be taken into the consideration, which helps
to better characterize the elevation information for the same
survey area [36]. LiDAR data have also been studied for
feature detection and extraction tasks [7], [37]–[39]. However,
in order to utilize the complementary information between HSI
and LiDAR data, a number of works have been proposed [40].
The joint use of HSI and LiDAR data can significantly
improve the classification performance by effectively address-
ing the shortcomings of each of them and compared to the indi-
vidual source of information [41]. In the beginning, most of
the researchers applied morphological extinction profiles (EPs)
and attribute profiles (APs) for joint feature extraction for
accurate classification of HSIs and LiDAR data [42]–[44].
Similarly, Rasti et al. [45] improved the joint extraction of
EPs by applying the total variation component analysis for
feature fusion. Merentitis et al. [46] introduced the automatic
fusion technique using the random forest classifier for joint
HSI and LiDAR data classification.
Nevertheless, nowadays, deep learning is being extensively
used for HSIs and LiDAR feature fusion [47] and classifica-
tion. An earlier attempt was made by considering LiDAR as
additional spectral bands of HSIs and fed together into a CNN
model to learn both the features and perform classification
tasks [48]. Ghamisi et al. [6] proposed a CNN model that
extracts traditional features, i.e., EPs jointly from HSIs and
LiDAR data that are fused and then fed to the CNN for
classification. Chen et al. [49] introduced an end-to-end fea-
ture fusion CNN model that can perform three different tasks
together, i.e., two separate CNNs are employed for feature
extraction directly from the HSIs and LiDAR data, and then,
extracted features were fused using concatenation and fed into
a fully connected (FC) layer for final classification.
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION
Likewise, Wu et al. [50] explored the spectral–spatial CNN
model for extracting discriminative HSI features and a spatial
CNN model to extract the information from another modality
data. Finally, both sources of information are concatenated,
and then, the fused features are classified using FC layers.
To reduce the network parameters, Hang et al. [8] proposed
an efficient HSI and LiDAR data fusion technique using a
shared CNN model and achieved performance gains compared
to the existing. To solve the spatially fragmented classification
problem, Zhao et al. [12] utilized hierarchical random walk in
the CNN layer to learn the spatial consistency between HSI
and LiDAR data for accurate classification. Hong et al. [51]
designed a novel semisupervised deep cross-modal network
by fully considering the unlabeled samples with the applica-
tion to the classification of multimodal remote sensing data.
Fang et al. [52] proposed an effective spatial–spectral cross-
modality enhancement module for joint HSI and LiDAR data
classification. Recently, Roy et al. [53] first introduced the
multimodal fusion transformer (MFT) model to learn the
abstract feature representation from both HSI and LiDAR data
by utilizing the classification (CLS) token for land-used and
land-cover classifications.
Morphology has been successfully applied in remote sens-
ing long back to extract spatial information based on the
concept of MPs and APs [54]–[57]. MPs and APs consist
of a number of handcrafted features (e.g., height, area, vol-
ume, diagonal of the bounding box, and standard deviation)
extracted by sequentially applying dilation and erosion oper-
ations with a set of structuring elements (SEs). Nevertheless,
MPs and APs do have a few common limitations. First, the
shape of the SE is fixed. Second, the SEs are only able
to extract information related to the size of existing objects
and fail to capture shape information of the arbitrary object
boundary for complex regions, especially for LiDAR data.
To overcome these limitations, a spatial morphological block
is introduced, which uses its own trainable SEs in dilation
and erosion layers to extract the robust and discriminative
elevation information from LiDAR data. At the same time, a
3-D CNN model is employed for extracting the spectral–spatial
features from HSIs. Finally, the two sources of features are
combined using concatenation operation to train both models
jointly in an end-to-end fashion. To summarize, the following
key contributions have been made in this article.
1) In order to take advantage of both the morphological
dilation and erosion operations, a novel spatial mor-
phological convolutional block with trainable kernels is
proposed to extract robust and discriminative elevation
information from LiDAR data.
2) The spectral–spatial features can be learned from HSI
data using a three-layered CNN that comprises Conv3D
layers, whereas the elevation information can be better
approximated from LiDAR data using trainable spatial
morphological blocks, and both features are concate-
nated using standard fusion, i.e., channel concatenation,
during training.
3) The spectral–spatial features from HSI data and the
elevation information from LiDAR data can efficiently
be learned using a joint training of a CNN and a spatial
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
5530416
morphological convolutional network for accurate clas-
sification of land covers.
4) Extensive experiments are conducted using two disjoint
datasets, i.e., University of Houston and Trento, where
the training-test samples are mutually exclusive, 5%
random training-test samples from the MUUFL dataset
are selected, and the proposed method performs best in
all three of them.
The rest of the article is organized as follows. The pre-
processing of HSIs and LiDAR data is described in Section II.
Section III introduces the proposed joint CNNs and the mor-
phological feature learning network. The experimental results
with detailed discussion are given in Section IV. Finally,
Section V concludes the article with some remarks and hints
at plausible future research lines.
II. P REPROCESSING OF HSI AND L I DAR DATA
Let us consider a spectral–spatial HSI XH ∈ R M×N ×B and
the corresponding LiDAR image XL ∈ R M×N where both
represent the same spatial region over the Earth’s surface.
Here, the width M and the height N, respectively, represent
two spatial dimensions for both images, and B refers to the
spectral bands of the HSI image.
All the pixels are classified into C land-cover classes
denoted by C = (y1 , y2 , . . . , yc ). The pixel xi, j ∈ XH ,
where i = 1, . . . , M and j = 1, . . . , N, and hence,
we can define the land-cover pixels as a spectral vector
xi, j = [x i, j,1 , . . . , x i, j,B ] ∈ R B containing B number of
elements. Similarly, to incorporate the spatial information,
patch extraction is performed in a preprocessing step, where
a spectral–spatial cube xi, j ∈ Rk×k×B ∈ XH from the
normalized HSI data XH and spatial cube xi, j ∈ Rk×k ∈ XL
from the LiDAR image with the neighboring regions of size
(k × k) are extracted, centered at the target pixel (i, j).
Jointly exploiting spatial–spectral information can increase
the discriminative power of feature learning network, and for
that reason, spectral–spatial cubes xh
(i, j )
are extracted from
raw HSIs and stacked into X H prior to the feature extraction
process. Similarly, for the LiDAR data, the spatial patch of
the same size xl
(i, j )
is extracted and stacked into X L before the
actual feature extraction process. Finally, the training and test
samples in each class for X H and X L can be represented by
 
D train = (x h , xl ), y (i) |i = 1, . . . , P
 
D test = (x h , xl ), y (i) |i = 1, . . . , Q (1)
where x h ∈ X H and xl ∈ X L are the samples randomly
chosen from HSIs and LiDAR data. P and Q represent the
number of training and test samples, and y (i) is the actual
class for the i th targeted land-cover pixels.
III. P ROPOSED M ETHODOLOGY
For better understanding, the core idea proposed (i.e., HSI +
LiDAR classification) in this article has been shown in Fig. 2.
The proposed HSI + LiDAR classification framework consists
of a three-layered 3-D CNN (Conv3D) for discriminative HSI
feature extraction and, similarly, a dual-branch morphological
2-D network to extract the robust spatial information from
5530416
Fig. 2. Graphical representation of the proposed hyperspectral and LiDAR data fusion network.
the LiDAR data. Both the extracted features are fused using
an addition ( ) operation immediately after the adaptive
average pooling layer, which is also followed by a classifier
that includes an FC layer with cross-entropy loss.
For the HSI, initially, channelwise normalization is per-
formed without affecting the spectral dimension of the HSI
data, and then, small cube patches of sizes 11 × 11 × B
are extracted surrounding the given target pixel. Similarly, the
image patches of the same spatial position as the HSI data are
directly extracted from LiDAR data. The goal is to effectively
combine both the information from XH and XL using a feature
fusion step to improve the classification performance.
A. Hyperspectral Feature Learning via 3-D CNNs
CNNs exhibit promising performance in large-scale image
classification tasks. The availability of enormous volumes of
HSIs enables us to utilize the benefits of CNNs. The HSI
XH often exhibits a nonlinear relationship between the spatial
information captured by the HSI sensors and the corresponding
object of HSI data. This type of nonlinear characteristic will
be more visible when dealing with XL .
The spectral–spatial behavior of HSIs always demands
spectral and spatial feature learning for discriminative and
robust classification. It has already been proved that CNNs
are capable to extract high-level abstract features, which are
also invariant to the source of data modalities, i.e., HSI and
LiDAR datasets.
To extract robust and discriminative spectral and spatial
features from raw HSIs, three layers of 3-D CNN are used
to model the network. Fig. 2 shows the overall graphical
representation of the Conv3D network for HSIs. The HSI
cube X H of size (11 × 11 × B) is passed through the three
consecutive Conv3D layers, and the sizes of the considered
kernels of the first two layers are chosen to be (3 × 3 × 7),
whereas the size of the used kernel in the last Conv3D layer
is (3 × 3 × 32).
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
The shapes of the output feature maps after the first and sec-
ond Conv3D layers are (11 × 11 × 69) and (11 × 11 × 32),
respectively. Similarly, the output of shape (11 × 11 × 1) is
produced by the last Conv3D layer and to keep the shape 1 in
the third dimension of the output feature maps after the last
Conv3D layer; the third dimension of the kernel is chosen to
be equal to the number feature maps, which is 32.
The convolutional layers utilize the feature padding mecha-
nism to ensure the same shape and spatial size for both input
and output feature maps. The batch normalization (BN) layer
is used after every Conv3D layer to regularize the problem
of overfitting due to the shortest of training samples and
accelerate the training performance. ReLU activation is used to
introduce the nonlinearities into the learning process during the
training of the network. The last two Conv3D layers employed
dropout in addition to the BN, which greatly reduces the
effects of the vanishing gradient problem.
B. LiDAR Feature Learning via a Morphological Network
Mathematical morphology is known to be powerful for
analyzing the intrinsic shape, structure, and size of the texture
object in the image. Here, a morphological network is designed
based on two primitive operations, i.e., dilation and erosion
using the widely used SEs of size (s × s) kernel. The channel
of LiDAR can be thought of as a surface over the high-
dimensional plane.
The input LiDAR patches are convolved with such kernels
to produce the dilated image, which contains maximum overall
pixels in its neighborhood defined locally. The dilation opera-
tion increases the boundaries of foreground pixels regions on
the input LiDAR image. Thus, the texture of various regions
in the LiDAR is enlarged depending on the size of the kernel.
Similarly, the output of the erosion operation contains the
minimum overall pixels of its local neighborhood during the
convolution with the kernel. Opposite to the dilation operation,
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION
erosion reduces the shape of the texture region in the LiDAR
image. It is worth noting that the erosion eliminates the minor
details, enlarges holes, and makes them separable between
different texture regions. Let X L ∈ R k×k be the input single
channel LiDAR patches of spatial size (k × k). We note that
the spatial sizes of both HSIs and LiDAR patches are kept
the same. The dilation and erosion operations can be denoted
by ⊕ and , respectively, since both the dilation and erosion
operations are nonlinear and produce the output by interacting
between the whole LiDAR image X L and a set called SEs,
which can be defined as follows:
(X L ⊕ Wd )(x, y) = max (X L (x + i, y + j ) + Wd (i, j )) (2)
(i, j )∈ψ
(X L  We )(x, y) = min (X L (x + i, y + j ) − We (i, j )) (3)
(i, j )∈ψ
where ψ = { (i, j ) | i ∈ {1, 2, 3, . . . , s}; j ∈ {1, 2, 3, . . . , s}}
represent the elements of the kernel, and Wd and We are
the kernels or SEs for the dilation and erosion operation,
respectively. It can be noted that the operations defined in (2)
and (3) are (s × s) window specific and closely similar with
the convolutional layers. To keep the input and output shape
same, the padding function is used after the dilation and
erosion operation. After applying (2) and (3) over the LiDAR
patch, the results are obtained as dilation map and erosion
map, respectively.
Fig. 3(a) and (b) describes the input and output images
after the dilation and erosion operations using defined SEs
of size 3 × 3 × 1. The dilation will enlarge the bright
regions while shrinking the dark ones of a particular tex-
ture feature in the morphological space. Similarly, erosion
will suppress the texture feature of SEs size. To obtain the
morphological shape feature from LiDAR data, a spatial
morphological (SpatialMorph) block (as shown in Fig. 2)
using the primitive operations is introduced in this article.
The proposed SpatialMorph block FMorph (X L , Wd , We , θ )
is derived using the (2) and (3) and given as follows:
(1,1)
FMorph (X L , Wd , We , θ ) = F2D ((X L ⊕ Wd ), θ )
(1,1)
 F2D ((X L  We ), θ ) (4)
where Wd and We are the weights of the (3 × 3) kernel
(1,1)
and F2D is the function that denotes the linear combination
between dilation and erosion feature maps obtained using 2-D
convolution parameterized with θ and kernels of size (1 × 1).
To obtain the resulting feature maps, a linear combination
of both dilation and erosion LiDAR features is calculated as
follows:
 IV. E XPERIMENTAL R ESULTS AND D ISCUSSION
(1,1)  int 
m
F2D X =b+ wl X int (x, y, l) (5)
l=1
where X int represents the intermediate dilation or erosion
feature maps, wl represents the weights for the lth feature
channel, and b represents the bias of the network. The SEs and
weights of the linear combination are initialized randomly.
C. Hyperspectral and LiDAR Feature Fusion
The spectral–spatial feature maps are extracted from X H
using 3-D CNNs. Similarly, the spatial and complementary
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
5530416
feature maps are also obtained from X L using morphological
CNN. To better learn both the complementary feature during
training of the network and the fusion of both the features
became an important issue for HSIs and LiDAR classification.
The traditional way to combine features from a different
source is to stack them and fuse the feature using the FC
layers. Due to the large size of the combined feature vector,
the linear layer often suffers from huge parameter phenomena,
which creates difficulty in training. To balance both the issues,
here, we use simple channelwise feature concatenation of both
the feature maps R H and R L immediately after the adaptive
average pooling layers

OHL = Fpool [F3D (X H , θ1 )]  Fpool FMorph (X L , θ2 ) (6)
where OHL ∈ R64×1×1 represent the fused output of the
obtained feature maps R H ∈ R32×1×1 and R L ∈ R32×1×1 .
Fpool is adaptive average ppooling operation, whereas X H and
X L are the input HSI and LiDAR data for the two baseline
networks, i.e., 3-D CNN (F3D ) and spatial morphological
CNN (FMorph ), which are parameterized with θ1 and θ2 ,
respectively.
Later, the obtained fused feature maps OHL are passed
through a flatten layer to produce a feature vector, which is
followed by a linear layer for accurate prediction of land-cover
classes. The output value of ŷ can be derived as follows:
ŷ = Fflatten (OHL , ω) = Flinear (ωOHL ) (7)
where ŷ ∈ RC×1 is the prediction, C is the number of
land-cover classes for the corresponding datasets, and Fflatten
is the linear layer parameterized with ω.
D. Training and Testing of the Joint Network
The proposed model (as shown in Fig. 2) is trained using
an end-to-end strategy where D train and D test represent the
training and testing sets, respectively. The training and test sets
contain P and Q numbers of labeled and unlabeled land-cover
samples for the training and validation of the proposed net-
work. The network is trained using the Adam [58], [59]
optimizer and backpropagates the training loss. The training
cross-entropy loss L CE between the predicted output ŷ (i) and
the targeted output y (i) can be calculated as follows:
1  (i)  (i)     
N
L CE = y log ŷ + 1 − y (i) log 1 − ŷ (i) . (8)
N i=1
A. Hyperspectral Datasets
To evaluate the performance of the proposed and compar-
ative networks, three different HSI scenes along with their
associated LiDAR data have been considered. The experimen-
tal datasets include the University of Houston (UH), Trento,
and MUUFL Gulfport (MUUFL) scenes. All these datasets are
further described in the following.
1) The IEEE Geoscience and Remote Sensing Society
published the University of Houston (UH) dataset—
collected by the Compact Airborne Spectrographic
5530416 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

Fig. 3. Graphical visualization of the dilation and erosion operations where an input image patch of size (7 × 7 × 1) is dilated and eroded with an SE of
size (3 × 3 × 1), and the obtained output size keeps the same with a padding mechanism. (a) Dilation Operation. (b) Erosion Operation.

Imager (CASI) in 2013, as part of its Data Fusion

Contest. The datasets consist of HSIs and LiDAR
data, and both are composed of 340 × 1905 pix-
els with 144 spectral bands. The spatial resolution of
this dataset is 2.5 MPP with a wavelength ranging
from 0.38–1.05 μm. Finally, the ground truth comprises
15 different land-cover classes. In addition, the available
samples are divided into disjoint training and test sam-
ples of 15 classes. The lists of disjoint training and test
samples for each of the 15 different land-cover classes
are shown in Fig. 4.
2) The MUUFL Gulfport scene was acquired by the Reflec-
tive Optics System Imaging Spectrometer (ROSIS)
sensor over the campus of University of Southern Mis-
sissippi Gulf Park, Long Beach Mississippi, in Novem-
ber 2010 [60], [61]. It consists of 325 × 220 pixels with
72 spectral bands, and the LiDAR modality consists of
elevation of two raster data. However, due to noise, the
initial and final eight bands are omitted leading to a total
of 64 bands. There are a total of 53 687 ground-truth
pixels describing 11 urban land-cover classes. Fig. 5
shows the list of 5% randomly chosen samples from
every 11 different classes.
3) The Trento scene was gathered by using the AISA eagle
sensor over the rural regions in the south of Trento, Italy,
while the LiDAR data were captured by the Optech Fig. 4. Visualization of the University of Houston (UH) scene. (a) Pseudo-
color image for the HSI data on bands 64, 43, and 22, respectively.
ALTM 3100EA sensor. The HSI contains 63 spectral (b) Grayscale image for the LiDAR data. (c) Ground truth of disjoint training
bands within a wavelength of range 0.42–0.99 μm, and samples. (d) Ground truth of disjoint test samples. The table represents class-
the LiDAR data has two rasters exploring elevation data. specific land-cover types and the number of disjoint training and test samples
i.e., 664 845 and 664 845, respectively. (a) HSI pseudocolor map. (b) LiDAR
The scene has 600 × 166 pixels that comprise six mutu- map. (c) Disjoint training samples. (d) Disjoint test sample.
ally exclusive vegetation land-cover classes where the
spectral resolution is 9.2 nm and the spatial resolution ground truth, the type associated with the land-cover classes,
is 1 MPP. In addition, the available samples are divided and the number of available labeled samples per class.
into disjoint training and test samples of six classes.
Fig. 6 lists the information about the per class number
of samples in six different land-cover classes. B. Experimental Protocols
In order to explore the effectiveness of the proposed net-
Figs. 4–6 show a detailed summary of the UH, MUUFL, work, several state-of-the-art methods have been compre-
and Trento scenes, respectively, including their corresponding hensively studied and compared. The comparative methods

Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION
Fig. 5. Visualization of the MUUFL scene. (a) True-color image for the
HSI data over bands 40, 20, and 10, respectively. (b) Grayscale image for the
LiDAR data. (c) Ground truth of MUUFL scene. The table represents class-
specific land-cover types and the number of randomly selected 5% training
and remaining 95% test samples, i.e., 71 500 and 71 500, respectively. (a) HSI
pseudocolor map. (b) LiDAR map. (c) Ground-truth map.
include both classical models and deep learning-based fea-
ture fusion models. The comparative methods include ran-
dom forest (RF) [62], SVM with radial basis function [15],
the recurrent neural network (RNN) [63], Two-CNN [50],
FusAtNet [64], CoupleNet [8], HWRN [12], HybridSN [33],
EndNet [65], and the proposed network.
The performance of the proposed network along with the
comparative methods is evaluated in terms of several widely
used quantitative measurements, such as overall accuracy
(OA), average accuracy (AA), and statistical Kappa (κ) coef-
ficient, respectively. OA is referred to the ratio of correctly
classified samples among the total test samples, whereas the
AA represents the mean of classwise accuracy. Moreover, κ
represents a strong mutual agreement between the generated
classification maps of the considered model and the provided
ground truth.
For experimental validation and to prove the superiority
of our proposed network, a five-cross-validation process has
been adopted and executed five times using 200 epochs per
iteration. The reported accuracies are the average result of
an aforesaid process. All the tests have been performed on a
Red Hat Enterprise Server (Release 7.6) with a CPU having
ppc64le architecture and a total of 40 cores with four threads
per core and 377 GB of RAM. The GPU used is a single
Nvidia Tesla V100 with 32 510 MB of VRAM. The source
code of the proposed MFT was implemented using PyTorch
1.5.0 and Python 3.7.7.
Moreover, the experimental process has been conducted in
three different settings.
1) Experiments have been conducted on disjoint (spatially
and spectrally) training and test samples, i.e., the inter-
section between the training samples and test samples
remains empty.
2) Different percentages of training samples have been used
to validate the performance of our proposed network.
3) Different spatial dimensions have also been considered
for experimental validation.
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
5530416
Fig. 6. Visualization of the Trento data. (a) True color image for the HSI
data over bands 40, 20, and 10, respectively. (b) Grayscale image for the
LiDAR data. (c) Ground truth of disjoint training samples. (d) Ground truth
of disjoint test samples. The table represents class-specific land-cover types
and the number of disjoint training and test samples, i.e., 99 600 and 99 600,
respectively. (a) HSI pseudocolor map. (b) LiDAR map. (c) Disjoint training
samples. (d) Disjoint test sample.
C. Performance Evaluation With Disjoint Train/Test Samples
In order to verify the performance of our proposed network,
a qualitative and quantitative comparison has been conducted
between the proposed method and several state-of-the-art
methods mentioned above. For a fair analysis and to make
the results reliable, each experiment has been repeated five
times; both the proposed and comparative methods have been
evaluated in the same experimental settings. The experimental
protocols are set as follows for the disjoint train/test samples
experiment. To process the HSI image, the spatial patch size
is set to (11 × 11 × B), where B is the number of bands for,
respectively, datasets, and the number of training epochs is
set to 200 without data augmentation and BN. The batch size
is set to 32, and the weights are optimized using the Adam
optimizer. Moreover, the learning rate of each method is set
as mentioned in their respective papers, whereas the learning
rate of our proposed method is set to 0.001. Finally, the
training and test samples’ percentages are set to 5%, i.e., 5%
randomly selected training samples, and the rest of the 95%
disjoint (spatially/spectrally) samples are unseen samples for
model evaluation.
The first kind of experiment is processed to check the
artifacts of disjoint training/test samples on different mod-
els, including the one proposed in this article. The disjoint
5530416 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

TABLE I
OA, AA, AND K APPA VALUES ON THE U NIVERSITY OF H OUSTON D ATASET ( IN %). “H” R EPRESENTS O NLY HSI, W HILE
“H + L” R EPRESENTS F USED HSI AND LiDAR

TABLE II
OA, AA, AND K APPA VALUES ON THE U NIVERSITY OF T RENTO D ATASET ( IN %). “H” R EPRESENTS O NLY HSI, W HILE
“H + L” R EPRESENTS F USED HSI AND LiDAR

TABLE III
OA, AA, AND K APPA VALUES ON THE MUUFL G ULFPORT D ATASET ( IN %). “H” R EPRESENTS O NLY HSI, W HILE “H + L”
R EPRESENTS F USED HSI AND LiDAR OVER 5% T RAINING S AMPLES

train/test samples significantly overcome the redundancy
among the seen and unseen samples, which is a prerequisite
condition of fair evaluation of any model. Usually, the remote
sensing researcher tends to use the randomly selected training
samples to build the model; however, the entire HSI datasets
are used at the testing time, which induces a strong bias.
Such a model does produce a higher accuracy but lacks
confidence especially when it comes to evaluating unseen
samples. Moreover, disjoint train/test samples also help to
avoid the overlapping region issues of HSI, especially at the
training time, i.e., the model has been evaluated on spectrally
and spatially disjoint samples, thus bringing more confidence.
The experimental results in terms of OA, AA, kappa (κ),
and classwise accuracies on disjoint train/test samples for
the University of Houston and Trento datasets are presented
in Tables I–II. On the other hand, Table III presents the
results on 5% randomly selected training samples (with the
rest of the samples used for testing) for the MUUFL dataset
as geographically disjoint train/test samples for MUUFL are
not available to the research community. Tables I–III show
results for both the HSI dataset and the fused HSI and LiDAR
datasets. The “H” column represents the results for the HSI
datasets (in the case of the proposed method, the LiDAR
branch is turned off), while the “H + L” column shows the
results for the fused HSI and LiDAR datasets (in the case of
the proposed method, both HSI branch and LiDAR branch
are turned on). It is evident that the overall performance
of almost all models improves with the use of LiDAR in
conjunction with HSI data compared to using HSI data alone.
Figs. 7–9 illustrate the classification maps for the University
of Houston, Trento, and MUUFL datasets, respectively. It can
be clearly seen that the classification maps for the proposed
model contain less salt and pepper noise compared to the clas-
sification maps of other models. From these results, one can
conclude that the proposed method (in both scenarios) works
better in almost 5% OA for HSI and 11% for HSI + LiDAR
case compared to the HybridSN; 7% for HSI and 10% for
HSI + LiDAR compared to HWRN; 9% for HSI and 11%

Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION 5530416

Fig. 7. (a) False color representation of the first PC obtained from the UH scene. Ground-truth classification maps along with the kappa (κ) accuracy
obtained for the DUH dataset for both HSI and combined HSI and LiDAR, (b) RF = 73.59 (HSI) and 78.93 (HSI + L), (c) SVM = = 67.02 (HSI) and
70.73 (HSI + L), (d) RNN = 72.05 (HSI) and 74.03 (HSI + L), (e) Two-CNN = 76.66 (HSI) and 85.59 (HSI + L), (f) FusAtNet = 78.06 (HSI) and 81.26
(HSI + L), (g) CoupledNet = 75.78 (HSI) and 79.78 (HSI + L), (h) HWRN = 77.08 (HSI) and 80.50 (HSI + L), (i) HybridSN = 79.70 (HSI) and 79.00
(HSI + L), (j) EndNet = 78.86 (HSI) and 82.63 (HSI + L), and (k) proposed model = 84.88 (HSI) and 90.55 (HSI + L). The higher accuracies are in bold
face.

Fig. 8. (a) False color representation of the first PC obtained from the Trento scene. Ground-truth (GT) classification maps along with the kappa (κ)
accuracy obtained for the DUH dataset for both HSI and combined HSI and LiDAR. (b) RF = 93.20 (HSI) and 97.51 (HSI + L), (c) SVM = 90.08
(HSI) and 92.59 (HSI + L), (d) RNN = 90.90 (HSI) and 91.49 (HSI + L), (e) Two-CNN = = 92.45 (HSI) and 97.37 (HSI + L), (f) FusAtNet = 89.66
(HSI) and 92.23 (HSI + L), (g) CoupledNet = 93.09 (HSI) and 96.87 (HSI + L), (h) HWRN = 92.95 (HSI) and 95.81 (HSI + L), (i) HybridSN = 92.12
(HSI) and 93.27 (HSI + L), (j) EndNet = 94.65 (HSI) and 96.21 (HSI + L), and (k) proposed model = 97.52 (HSI) and 98.71 (HSI + L). The higher
accuracies are in bold face.

for HSI + LiDAR compared to CoupledNet; 6% for HSI and
9% for HSI + LiDAR compared to FusAtNet; 4% for HSI
and 5% for HSI + LiDAR compared to Two-CNN; 8% for
HSI and 16% for HSI + LiDAR compared to RNN; 17% for
HSI and 20% for HSI + LiDAR compared to SVM; and 13%
for HSI and 12% for HSI + LiDAR compared to RF. Though
some methods achieve superior classwise accuracies using just
HSI in some cases, the overall performance using fused HSI
and LiDAR is far better. Similar observations can be made for
the other experimental datasets in which the proposed network
outperformed state-of-the-art and conventional methods.
It has been observed that both convolution, and morpho-
logical dilation and erosion are the neighborhood operators
with respect to the trainable kernels [66], [67]. To extract
the features from HSI data, the convolutional layer uses a set
of trainable filters that allow performing linear combinations
between the HSI cubes and the weights of the kernel. However,
it is not enough for HSI data to distinguish between two
different classes by considering only the spectral information
since both the classes produce similar spectral responses.
To provide complementary information to the HSI data,
LiDAR data were considered, which successfully captured the
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
elevation information of each class in an image, such as height,
width, and shape, covering the same surveyed area. Unlike
convolution operations, morphological operators are nonlinear
operations in nature that can better approximate the shape
information of the image classes.
To achieve this, the morphological layer (i.e., dilation
and erosion) replaces the linear convolution operation with
trainable morphological max or min operators, which allows
the proposed network to capture distinct and robust eleva-
tion information from LiDAR data compared to conventional
CNNs. Both the dilation and erosion feature maps preserve
the nonlinear structure of the LiDAR data and are parallelly
learned to aggregate the most important nonlinear information
from LiDAR data, and both the feature maps are normalized
using a 2-D convolution operation with a kernel of size
(1 × 1), which reduces the feature discrepancy obtained from
dilation and erosion layers.
However, the LiDAR data also fail to discriminate between
two different roads even though roads are formed using
different materials, but both roads produced similar elevation
information. Therefore, it is a good choice to combine both
HSI and LiDAR information, which provides complementary
5530416 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

Fig. 9. (a) False color representation of the first PC obtained from the MUUFL scene. Ground-truth (GT) classification maps along with the kappa (κ)
accuracy obtained for the DUH dataset for both HSI and combined HSI and LiDAR. (b) RF = 86.88 (HSI) and 87.01 (HSI + L), (c) SVM = 74.33 (HSI)
and 72.89 (HSI + L), (d) RNN = 85.86 (HSI) and 86.47 (HSI + L), (e) Two-CNN = 89.53 (HSI) and 90.79 (HSI + L), (f) FusAtNet = 88.97 (HSI) and
90.18 (HSI + L), (g) CoupledNet = 88.89 (HSI) and 89.93 (HSI + L), (h) HWRN = 90.94 (HSI) and 90.25 (HSI + L), (i) HybridSN = 88.62 (HSI) and
90.19 (HSI + L), (j) EndNet = 78.36 (HSI) and 82.17 (HSI + L), and (k) proposed model = 90.90 (HSI) and 91.33 (HSI + L). The higher accuracies are
in bold faces.

information to each other and can significantly improve clas-
sification performance. In the above context, we employed
morphological dilation and erosion layers for their usefulness
to learn nonlinear object boundaries for complex classes from
LiDAR data. The fused information of HSI and LiDAR
outperformed the existing results in terms of OA, AA, and
Kappa coefficient.
D. Performance Evaluation With Different Spatial
Dimensions
The classification performance of any spectral–spatial net-
work is also dependent on the spatial dimension of the input
cubes. Thus, the patch size has an important impact on the
result. Therefore, this section presents several experimental
results with different patch sizes to process HSI cubes to
explore the impact of different patch sizes on classification
results. In all these experiments, the initial training/test sam-
ples percentages are set as explained in the previous sections.
All other training parameters remain the same except the patch
sizes that are tested as 7 × 7, 9 × 9, and 11 × 11. The OA,
AA, and κ accuracies have been considered and presented in
Table IV to validate the performance of the proposed network
along with the comparative methods on different patch sizes.
The optimal results are highlighted in bold across all the spatial
sizes per dataset.
The superiority of the proposed model across various spatial
windows is also quite evident from Table IV. The proposed
model achieves 97.29% and 99.70% accuracies both over the
11 × 11 and 94.16% over 7 × 7 in terms of OAs across all the
windows for UH, Trento, and MUUFL datasets respectively.
Similarly, AAs of 97.35% and 99.31% are achieved by the
proposed network using 11 × 11 and 7 × 7, respectively, for
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
both the UH and Trento datasets, whereas FusAtNet shows
its prowess in the MUUFL dataset over the 11 × 11 window.
Finally, the proposed network achieves 97.07% and 99.61%
of Kappa values for UH and Trento datasets over the window
of size 11 × 11 and 92.27% for the MUUFL dataset over
9 × 9, and dominates all other models. HWRN can only
achieve better AA (83.12%) than the proposed model for
the MUUFL dataset across all the spatial windows. All deep
learning methods (Two-CNN, FusAtNet, CoupledNet, HWRN,
HybridSN, and EndNet) perform competitively but cannot
achieve the results provided by the proposed model.
From Table IV, one can find that the classification per-
formance significantly improves as the spatial patch size
increases. As a matter of fact, the larger the patch size, the
most information can be characterized, thus producing better
accuracies. The fact remains the same for all experimental
datasets until the patch contains the interfering samples. Since,
if some interfering samples appeared in the same patch, then
the accuracies may decrease significantly. Therefore, a suitable
patch size is very crucial for appropriate classification results.
E. Performance Evaluation With Different Numbers of
Training Samples
Despite the aforesaid experiments, there is another challenge
that needs to be evaluated to validate the claims, i.e., the
number of labeled training samples. If the percentage of the
training samples is not adequate or not reliable, it may lead
to the under-fitting (also known as the Hughes phenomena)
or overfitting (not reliable) issues. Thus, how to select an
appropriate number of training samples is another important
factor for classification accuracy. This section presents rigor-
ous experimental results on different percentages of training
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION 5530416

TABLE IV
OVERALL (OA), AVERAGE (AA), AND K APPA (κ ) A CCURACIES (%) OF RF, SVM, RNN, T WO -CNN, FusAtNet, CoupledNet, HWRN, HybridSN, AND
THE P ROPOSED N ETWORK U SING D IFFERENT S PATIAL W INDOW S IZES , I . E ., 7 × 7, 9 × 9, AND 11 × 11, R ESPECTIVELY

Fig. 10. Overall accuracy (OA) achieved by different methods with varying training sample sizes that are randomly taken from (a) UH, (b) UT, and
(c) MUUFL datasets.

samples, i.e., 5%, 8%, 10%, and 12%, respectively, are ran-
domly selected to train the models (both the one proposed in
this article and the comparative methods), and the rest of the
samples are used for testing purposes. The rest of the parame-
ters remain the same as explained in the previous sections.
The detailed experimental results are presented in
Figs. 10–12. From the figures, one can conclude that the
experimental results obtained on 5% training samples are
inferior to the ones obtained with 12% training samples on
all three datasets with all the methods. Figs. 10–12 show
that the detailed classification performance using OA, AA,
and κ metrics on three datasets with different percentages of
randomly selected training samples.
Figs. 10–12 show that the classification accuracies signifi-
cantly improve as the number of training samples increases,
which proves that the spatial–spectral feature learning process
proposed in this study is effective for HSI classification
while considering the limited availability of training samples.
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
Achieving such higher accuracy and statistical significance
while with a limited number of training samples is cru-
cial, especially for HSI classification since acquiring a high
number of labeled training samples is often difficult and
time-consuming in real scenarios.
F. Ablation Study Along With Statistical Tests
In order to show that the performance of the proposed model
is significant and outperformed compared to the state-of-the-
art methods, we have used McNemar’s statistical significance
test [68]. McNemar’s test is used to identify whether the
experimental results obtained are significantly better or not.
Moreover, it is being used to evaluate the statistical signifi-
cance differences for a sample set, which is calculated based
on the standardized normal statistic test
f 12 − f21
Z= √ (9)
f 12 + f 21
5530416 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

Fig. 11. Average accuracy (AA) achieved by different methods with varying training sample sizes that are randomly taken from (a) UH, (b) UT, and
(c) MUUFL datasets.

Fig. 12. Kappa (κ) accuracy achieved by different methods with varying training sample sizes that are randomly taken from (a) UH, (b) UT, and (c) MUUFL
datasets.

Fig. 13. 2-D feature visualization of the proposed joint spectral–spatial HSIs and LiDAR features via the t-SNE method for disjoint UH and Trento dataset.
(a) HSIs. (b) HSIs + LiDAR. (c) HSIs. (d) HSIs + LiDAR.

where f 12 indicates the number of samples correctly classified
by classifier 1 and wrongly classified by the classifier 2.
Similarly, f 21 is the number of samples that can be correctly
classified by method 2 but wrongly classified by method 1.
The difference between the two methods is said to be
statistically significant if |Z | > 1.96, whereas Z > 0 indicates
that method 1 is more discriminative than method 2, and
Z < 0 means that method 2 is more discriminative than
method 1.
Fig. 13 shows the t-distributed stochastic neighbor embed-
ding (t-SNE) process to visualize the 2-D feature spaces
learned using the proposed method. t-SNE is usually used to
visualize the nonlinear connections among the data samples
and, moreover, shows the capacity of any model to learn
the nonlinear. t-SNE is relatively helpful when dealing with
CNN or any other deep network because deep architectures
are usually considered a black box, and there is no way to
interpret what is happening at deeper levels of the architecture.
Generically, we assume that the deeper level contains the
information about more complex objects; however, to some
extent, this is not completely true. One can interpret it as
explained above; however, HSI datasets are themselves just
a high-dimensional noise for individuals [69]. Thus, t-SNE
can significantly help us to understand which samples seem
similar for the deep model.
Furthermore, the statistical significance values between the
proposed method and the compared methods are shown in
Table V using the disjoint training samples, i.e., UH and Trento
datasets and 5% of training samples for the MUUFL dataset.
It is observed that the proposed model performs significantly
better for the disjoint datasets compared to the MUUFL
dataset, which uses 5% of randomly selected training samples.

Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION 5530416

TABLE V
S TATISTICAL S IGNIFICANCE OF D IFFERENCE (Z VALUE ) B ETWEEN D IFFERENT M ETHODS OVER HSI S AND LiDAR D ATASETS

TABLE VI
OA, AA, AND K APPA VALUES ( IN %) ON UH, T RENTO , AND MUUFL D ATASETS FOR THE P ROPOSED M ODEL U SING THE VARIOUS HSI AND LiDAR
D ATA F USION S TRATEGY, I . E ., E ARLY, M IDDLE , L ATE , AND C ROSS F USIONS

Moreover, here, we provide an ablation study to further
validate the fusion process by using a recently proposed
idea [41] related to what, where, and how to fuse, i.e., four dif-
ferent fusion strategies (such as “early,” “middle,” “late,” and
“cross” fusion strategies) have been used. More specifically,
the fusion process is performed after feature learning using our
proposed network. Table VI listed the quantitative comparison
between different fusion strategies in terms of OA, AA, and
Kappa on HU, Trento, and MUUFL datasets. In general, the
feature representations from Hyperspectral and LiDAR data
obtained by our proposed feature learning network tend to
be fused using the “early” fusion strategy, yielding slightly
better classification results on all the experimental datasets
compared to those using “middle” and “later” fusion strategies.
This possible reason for the phenomenon is that the learned
features via the proposed approach are discriminative and
“good” enough, and there is no need to use more advanced
and complex fusion strategies. The cross-fusion mechanism is
well designed in [41] to address the problem of cross-modality
learning, which is an end-to-end combination framework,
including feature learning and fusion. It should be noted,
however, that the proposed feature learning network is more
powerful in learning feature representations. This, to some
extent, can explain why the fusion results obtained by the
cross-fusion strategy are relatively worse than other more
simple ones, e.g., “early,” “middle,” and “late” fusion.
V. C ONCLUSION AND F UTURE W ORK
A new joint CNN and morphological feature learning frame-
work for HSI and LiDAR data fusion is proposed for accurate
land-cover classification. The proposed network learns joint
feature representations for HSI and LiDAR data for classifica-
tion purposes. The proposed model consists of a three-layered
CNN and a morphological convolutional network in which
the former uses 3-D convolution to effectively extract the
spectral–spatial features from HSI data, and the latter can
effectively model the complimentary and nonlinear features
from LiDAR data by considering morphological dilation and
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
erosion layers, followed by a 2-D convolution with a kernel
of size (1 × 1). To aggregate the most important nonlinear
and elevation information, both dilation and erosion layers are
learned in parallel. Finally, the spectral–spatial features from
HSI data and elevation information from LiDAR data are fused
for robust classification. To validate the performance of the
proposed network, two datasets, i.e., UH and UT with disjoint
training and test samples, and randomly selected training and
test samples from MUUFL have been considered for the exper-
imental evaluation. The results show that the proposed model
achieves significant performance improvements in terms of
OA, AA, and Kappa for both UH, UT, and MUUFL datasets.
Future works will focus on developing parallel implementa-
tions of the proposed approaches for faster exploitation. In the
future, we will also consider the use of more sophisticated
morphological filters (such as extended morphological profiles
or attribute profiles) instead of the simple dilation and erosion
operations for morphological feature extraction.
ACKNOWLEDGMENT
The authors would like to thank Prof. Lorenzo Bruzzone,
University of Trento, Trento, Italy, for providing the Trento
Data. They also thank Ganesan Narayanasamy who is leading
IBM OpenPOWER/POWER enablement and ecosystem
worldwide for his support to get the IBM AC922 system’s
access.
R EFERENCES
[1] M. Ahmad et al., “Hyperspectral image classification—Traditional to
deep models: A survey for future prospects,” IEEE J. Sel. Topics
Appl. Earth Observ. Remote Sens., vol. 15, pp. 968–999, 2022, doi:
10.1109/JSTARS.2021.3133021.
[2] D. Hong et al., “Interpretable hyperspectral artificial intelligence: When
nonconvex modeling meets hyperspectral remote sensing,” IEEE Geosci.
Remote Sens. Mag., vol. 9, no. 2, pp. 52–87, Jun. 2021.
[3] P. O. Gislason, J. A. Benediktsson, and J. R. Sveinsson, “Random Forests
for land cover classification,” Pattern Recognit. Lett., vol. 27, no. 4,
pp. 294–300, 2006.
[4] B. Rasti et al., “Feature extraction for hyperspectral imagery: The
evolution from shallow to deep: Overview and toolbox,” IEEE Geosci.
Remote Sens. Mag., vol. 8, no. 4, pp. 60–88, Dec. 2020.
5530416
[5] P. Ghosh, S. Kumar Roy, B. Koirala, B. Rasti, and P. Scheun-
ders, “Deep hyperspectral unmixing using transformer network,” 2022,
arXiv:2203.17076.
[6] P. Ghamisi, B. Höfle, and X. X. Zhu, “Hyperspectral and LiDAR data
fusion using extinction profiles and deep convolutional neural network,”
IEEE J. Sel. Topics Appl. Earth Observ. Remote Ses., vol. 10, no. 6,
pp. 3011–3024, Jun. 2017.
[7] B. Höfle, M. Hollaus, and J. Hagenauer, “Urban vegetation detection
using radiometrically calibrated small-footprint full-waveform airborne
lidar data,” ISPRS J. Photogramm. Remote Sens., vol. 67, pp. 134–147,
Jan. 2012.
[8] R. Hang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, “Classification
of hyperspectral and LiDAR data using coupled CNNs,” IEEE Trans.
Geosci. Remote Sens., vol. 58, no. 7, pp. 4939–4950, Jul. 2020.
[9] M. Ahmad et al., “Spatial prior fuzziness pool-based interactive clas-
sification of hyperspectral images,” Remote Sens., vol. 11, no. 9,
p. 1136, May 2019. [Online]. Available: https://www.mdpi.com/2072-
4292/11/9/1136
[10] M. Ahmad, S. Shabbir, D. Oliva, M. Mazzara, and
S. Distefano, “Spatial-prior generalized fuzziness extreme learning
machine autoencoder-based active learning for hyperspectral image
classification,” Optik, vol. 206, Mar. 2020, Art. no. 163712.
[Online]. Available: https://www.sciencedirect.com/science/
article/pii/S0030402619316109
[11] D. Hong et al., “SpectralFormer: Rethinking hyperspectral image classi-
fication with transformers,” IEEE Trans. Geosci. Remote Sens., vol. 60,
pp. 1–15, 2022, doi: 10.1109/TGRS.2021.3130716.
[12] X. Zhao, R. Tao, and W. Li, “Multisource remote sensing data clas-
sification using deep hierarchical random walk networks,” in Proc.
IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2019,
pp. 2187–2191.
[13] M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, M. Ali, and
M. S. Sarfraz, “A fast and compact 3-D CNN for hyperspectral image
classification,” IEEE Geosci. Remote Sens. Lett., vol. 19, pp. 1–5, 2022.
[14] D. Hong, J. Yao, D. Meng, Z. Xu, and J. Chanussot, “Multimodal GANs:
Toward crossmodal hyperspectral–multispectral image segmentation,”
IEEE Trans. Geosci. Remote Sens., vol. 59, no. 6, pp. 5103–5113,
Jun. 2021.
[15] F. Melgani and L. Bruzzone, “Classification of hyperspectral remote
sensing images with support vector machines,” IEEE Trans. Geosci.
Remote Sens., vol. 42, no. 8, pp. 1778–1790, Aug. 2004.
[16] M. Ahmad et al., “Multiclass non-randomized spectral–spatial active
learning for hyperspectral image classification,” Appl. Sci., vol. 10,
no. 14, p. 4739, 2020. [Online]. Available: https://www.mdpi.com/2076-
3417/10/14/4739
[17] D. Hong, N. Yokoya, J. Chanussot, J. Xu, and X. X. Zhu, “Joint and
progressive subspace analysis (JPSA) with spatial–spectral manifold
alignment for semisupervised hyperspectral dimensionality reduction,”
IEEE Trans. Cybern., vol. 51, no. 7, pp. 3602–3615, Jul. 2021.
[18] S. Zhang, X. Kang, P. Duan, B. Sun, and S. Li, “Polygon structure-
guided hyperspectral image classification with single sample for strong
geometric characteristics scenes,” IEEE Trans. Geosci. Remote Sens.,
vol. 60, pp. 1–12, 2022.
[19] P. Duan, P. Ghamisi, X. Kang, B. Rasti, S. Li, and R. Gloaguen, “Fusion
of dual spatial information for hyperspectral image classification,” IEEE
Trans. Geosci. Remote Sens., vol. 59, no. 8, pp. 7726–7738, Sep. 2020.
[20] X. Kang, S. Li, L. Fang, and J. A. Benediktsson, “Intrinsic image
decomposition for feature extraction of hyperspectral images,” IEEE
Trans. Geosci. Remote Sens., vol. 53, no. 4, pp. 2241–2253, Apr. 2015.
[21] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., 2012, pp. 1097–1105.
[22] H. Lee and H. Kwon, “Going deeper with contextual CNN for hyper-
spectral image classification,” IEEE Trans. Image Process., vol. 26,
no. 10, pp. 4843–4855, Oct. 2017.
[23] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph
convolutional networks for hyperspectral image classification,” IEEE
Trans. Geosci. Remote Sens., vol. 59, no. 7, pp. 5966–5978, Jul. 2021.
[24] S. K. Roy, J. M. Haut, M. E. Paoletti, S. R. Dubey, and A. Plaza, “Gen-
erative adversarial minority oversampling for spectral–spatial hyperspec-
tral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60,
2021, Art. no. 5500615.
[25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 770–778.
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
[26] Z. Zhong, J. Li, Z. Luo, and M. Chapman, “Spectral–spatial residual
network for hyperspectral image classification: A 3-D deep learn-
ing framework,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 2,
pp. 847–858, Feb. 2017.
[27] S. K. Roy, S. Chatterjee, S. Bhattacharyya, B. B. Chaudhuri, and
J. Platoš, “Lightweight spectral–spatial squeeze-and-excitation residual
bag-of-features learning for hyperspectral classification,” IEEE Trans.
Geosci. Remote Sens., vol. 58, no. 8, pp. 5277–5290, Aug. 2020.
[28] S. K. Roy, S. Manna, T. Song, and L. Bruzzone, “Attention-based adap-
tive spectral–spatial kernel ResNet for hyperspectral image classifica-
tion,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 9, pp. 7831–7843,
Sep. 2021.
[29] M. E. Paoletti, J. M. Haut, R. Fernandez-Beltran, J. Plaza, A. J. Plaza,
and F. Pla, “Deep pyramidal residual networks for spectral–spatial
hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens.,
vol. 57, no. 2, pp. 740–754, Feb. 2019.
[30] S. K. Roy, D. Hong, P. Kar, X. Wu, X. Liu, and D. Zhao, “Lightweight
heterogeneous kernel convolution for hyperspectral image classification
with noisy labels,” IEEE Geosci. Remote Sens. Lett., vol. 19, pp. 1–5,
2022.
[31] S. K. Roy, P. Kar, D. Hong, X. Wu, A. Plaza, and J. Chanussot,
“Revisiting deep hyperspectral feature extraction networks via gradient
centralized convolution,” IEEE Trans. Geosci. Remote Sens., vol. 60,
pp. 1–19, 2022.
[32] S. K. Roy, M. E. Paoletti, J. M. Haut, E. M. T. Hendrix, and
A. Plaza, “A new max-min convolutional network for hyperspectral
image classification,” in Proc. 11th Workshop Hyperspectral Imag.
Signal Process., Evol. Remote Sens. (WHISPERS), Mar. 2021, pp. 1–5.
[33] S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri, “HybridSN:
Exploring 3-D–2-D CNN feature hierarchy for hyperspectral image clas-
sification,” IEEE Geosci. Remote Sens. Lett., vol. 17, no. 2, pp. 277–281,
Jun. 2020.
[34] M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, S. K. Roy,
and X. Wu, “Hybrid dense network with attention mechanism
for hyperspectral image classification,” IEEE J. Sel. Topics Appl.
Earth Observ. Remote Sens., vol. 15, pp. 3948–3957, 2022, doi:
10.1109/JSTARS.2022.3171586.
[35] S. Kumar Roy, R. Mondal, M. E. Paoletti, J. M. Haut, and A. Plaza,
“Morphological convolutional neural networks for hyperspectral image
classification,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.,
vol. 14, pp. 8689–8702, 2021.
[36] C. Debes et al., “Hyperspectral and LiDAR data fusion: Outcome of
the 2013 GRSS data fusion contest,” IEEE J. Sel. Topics Appl. Earth
Observ. Remote Sens., vol. 7, no. 6, pp. 2405–2418, Jun. 2014.
[37] M. Rutzinger, B. Höfle, M. Hollaus, and N. Pfeifer, “Object-based
point cloud analysis of full-waveform airborne laser scanning data for
urban vegetation classification,” Sensors, vol. 8, no. 8, pp. 4505–4528,
Aug. 2008.
[38] M. Dalponte, L. Bruzzone, and D. Gianelle, “Fusion of hyperspectral
and LIDAR remote sensing data for classification of complex forest
areas,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp. 1416–1427,
May 2008.
[39] R. M. Lucas, A. C. Lee, and P. J. Bunting, “Retrieving forest biomass
through integration of CASI and LiDAR data,” Int. J. Remote Sens.,
vol. 29, no. 5, pp. 1553–1577, Mar. 2008.
[40] P. Ghamisi et al., “Multisource and multitemporal data fusion in remote
sensing: A comprehensive review of the state of the art,” IEEE Geosci.
Remote Sens. Mag., vol. 7, no. 1, pp. 6–39, Mar. 2019.
[41] D. Hong et al., “More diverse means better: Multimodal deep learn-
ing meets remote-sensing imagery classification,” IEEE Trans. Geosci.
Remote Sens., vol. 59, no. 5, pp. 4340–4354, May 2021.
[42] M. Dalla Mura, J. A. Benediktsson, B. Waske, and L. Bruzzone,
“Morphological attribute profiles for the analysis of very high reso-
lution images,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 10,
pp. 3747–3762, Oct. 2010.
[43] W. Liao, R. Bellens, A. Pizurica, S. Gautama, and W. Philips,
“Graph-based feature fusion of hyperspectral and LiDAR remote
sensing data using morphological features,” in Proc. IGARSS, 2013,
pp. 4942–4945.
[44] P. Ghamisi, J. A. Benediktsson, and S. Phinn, “Land-cover classification
using both hyperspectral and LiDAR data,” Int. J. Image Data Fusion,
vol. 6, no. 3, pp. 189–215, 2015.
[45] B. Rasti, P. Ghamisi, and R. Gloaguen, “Hyperspectral and LiDAR
fusion using extinction profiles and total variation component analysis,”
IEEE Trans. Geosci. Remote Sens., vol. 55, no. 7, pp. 3997–4007,
Jul. 2017.
ROY et al.: HYPERSPECTRAL AND LiDAR DATA CLASSIFICATION
[46] A. Merentitis, C. Debes, R. Heremans, and N. Frangiadakis, “Auto-
matic fusion and classification of hyperspectral and LiDAR data using
random forests,” in Proc. IEEE Geosci. Remote Sens. Symp., Jul. 2014,
pp. 1245–1248.
[47] L. Gómez-Chova, D. Tuia, G. Moser, and G. Camps-Valls, “Multimodal
classification of remote sensing images: A review and future directions,”
Proc. IEEE, vol. 103, no. 9, pp. 1560–1584, Aug. 2015.
[48] S. Morchhale, V. P. Pauca, R. J. Plemmons, and T. C. Torgersen, “Clas-
sification of pixel-level fused hyperspectral and lidar data using deep
convolutional neural networks,” in Proc. 8th Workshop Hyperspectral
Image Signal Process., Evol. Remote Sens. (WHISPERS), Aug. 2016,
pp. 1–5.
[49] Y. Chen, C. Li, P. Ghamisi, X. Jia, and Y. Gu, “Deep fusion of remote
sensing data for accurate classification,” IEEE Geosci. Remote Sens.
Lett., vol. 14, no. 8, pp. 1253–1257, Aug. 2017.
[50] X. Wu, D. Hong, and J. Chanussot, “Convolutional neural net-
works for multimodal remote sensing data classification,” IEEE
Trans. Geosci. Remote Sens., vol. 60, pp. 1–10, 2022, doi:
10.1109/TGRS.2021.3124913.
[51] D. Hong, N. Yokoya, G.-S. Xia, J. Chanussot, and X. X. Zhu,
“X-ModalNet: A semi-supervised deep cross-modal network for clas-
sification of remote sensing data,” ISPRS J. Photogramm. Remote Sens.,
vol. 167, pp. 12–23, Sep. 2020.
[52] S. Fang, K. Li, and Z. Li, “S2 ENet: Spatial-spectral cross-modal
enhancement network for classification of hyperspectral and LiDAR
data,” IEEE Geosci. Remote Sens. Lett., vol. 19, 2021, Art. no. 6504205.
[53] S. Kumar Roy, A. Deria, D. Hong, B. Rasti, A. Plaza, and J. Chanussot,
“Multimodal fusion transformer for remote sensing image classification,”
2022, arXiv:2203.16952.
[54] M. Pesaresi and J. A. Benediktsson, “A new approach for the morpho-
logical segmentation of high-resolution satellite imagery,” IEEE Trans.
Geosci. Remote Sens., vol. 39, no. 2, pp. 309–320, Feb. 2001.
[55] M. Pedergnana, P. R. Marpu, M. D. Mura, J. A. Benediktsson, and
L. Bruzzone, “Classification of remote sensing optical and LiDAR data
using extended attribute profiles,” IEEE J. Sel. Topics Signal Process.,
vol. 6, no. 7, pp. 856–865, Nov. 2012.
[56] D. Hong, X. Wu, P. Ghamisi, J. Chanussot, N. Yokoya, and X. X. Zhu,
“Invariant attribute profiles: A spatial-frequency joint feature extractor
for hyperspectral image classification,” IEEE Trans. Geosci. Remote
Sens., vol. 58, no. 6, pp. 3791–3808, Jun. 2020.
[57] S. K. Roy, B. Chanda, B. B. Chaudhuri, D. K. Ghosh, and S. R. Dubey,
“Local morphological pattern: A scale space shape descriptor for texture
classification,” Digit. Signal Process., vol. 82, pp. 152–165, Nov. 2018.
[58] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
2014, arXiv:1412.6980.
[59] S. R. Dubey, S. Chakraborty, S. K. Roy, S. Mukherjee, S. K. Singh, and
B. B. Chaudhuri, “DiffGrad: An optimization method for convolutional
neural networks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 11,
pp. 4500–4511, Nov. 2020.
[60] P. Gader, A. Zare, R. Close, J. Aitken, and G. Tuell, “Muufl gulfport
hyperspectral and lidar airborne data set,” Univ. Florida, Gainesville,
FL, USA, Tech. Rep. REP-2013-570, 2013.
[61] X. Du and A. Zare, “Scene label ground truth map for muufl gulfport
data set,” Dept. Elect. Comput. Eng., Univ. Florida, Gainesville, FL,
USA, Tech. Rep., 2017.
[62] D. Hong, J. Hu, J. Yao, J. Chanussot, and X. X. Zhu, “Multimodal
remote sensing benchmark datasets for land cover classification with
a shared and specific feature learning model,” ISPRS J. Photogramm.
Remote Sens., vol. 178, pp. 68–80, Aug. 2021.
[63] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the
properties of neural machine translation: Encoder-decoder approaches,”
2014, arXiv:1409.1259.
[64] S. Mohla, S. Pande, B. Banerjee, and S. Chaudhuri, “FusAtNet: Dual
attention based SpectroSpatial multimodal fusion network for hyperspec-
tral and LiDAR classification,” in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit. Workshops (CVPRW), Jun. 2020, pp. 92–93.
[65] D. Hong, L. Gao, R. Hang, B. Zhang, and J. Chanussot, “Deep encoder-
decoder networks for classification of hyperspectral and LiDAR data,”
IEEE Geosci. Remote Sens. Lett., vol. 19, 2022, Art. no. 5500205, doi:
10.1109/LGRS.2020.3017414.
[66] K. Nogueira, J. Chanussot, M. D. Mura, and J. A. D. Santos, “An
introduction to deep morphological networks,” IEEE Access, vol. 9,
pp. 114308–114324, 2021.
[67] R. Mondal, P. Purkait, S. Santra, and B. Chanda, “Morphological
networks for image de-raining,” in Proc. Int. Conf. Discrete Geometry
Comput. Imag. Marne-la-Vallée, France: Springer, 2019, pp. 262–275.
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
5530416
[68] G. M. Foody, “Thematic map comparison,” Photogramm. Eng. Remote
Sens., vol. 70, no. 5, pp. 627–633, May 2004.
[69] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “An augmented linear
mixing model to address spectral variability for hyperspectral unmixing,”
IEEE Trans. Image Process., vol. 28, no. 4, pp. 1923–1938, Apr. 2019.
Swalpa Kumar Roy (Student Member, IEEE)
received the bachelor’s degree from the West Bengal
University of Technology, Kolkata, India, in 2012,
the master’s degree from the Indian Institute of Engi-
neering Science and Technology, Shibpur, Howrah,
India, in 2015, and the Ph.D. degree from the Uni-
versity of Calcutta, Kolkata, in 2021, all in computer
science and engineering.
From July 2015 to March 2016, he was a Project
Linked Person with the Optical Character Recogni-
tion (OCR) Laboratory, Computer Vision and Pattern
Recognition Unit, Indian Statistical Institute, Kolkata. He is currently working
as an Assistant Professor with the Department of Computer Science and Engi-
neering, Jalpaiguri Government Engineering College, Jalpaiguri, West Bengal,
India. His research interests include computer vision, deep learning, and
remote sensing.
Dr. Roy was nominated for the Indian National Academy of Engineer-
ing (INAE) Engineering Teachers Mentoring Fellowship Program by INAE
Fellows in academic tenure 2021–2022 and also a recipient of the Outstanding
Paper Award in second Hyperspectral Sensing Meets Machine Learning and
Pattern Analysis (HyperMLPA) at the Workshop on Hyperspectral Imaging
and Signal Processing: Evolution in Remote Sensing (WHISPERS) in 2021.
He has served as a Reviewer for the IEEE T RANSACTIONS ON G EOSCIENCE
AND R EMOTE S ENSING and IEEE G EOSCIENCE AND R EMOTE S ENSING
L ETTERS .
Ankur Deria is currently pursuing the B.Tech.
degree with the Department of Computer Science
and Engineering, Jalpaiguri Government Engineer-
ing College, Jalpaiguri, India.
His research interests include computer vision and
deep learning.
Mr. Deria was nominated for the Indian National
Academy of Engineering (INAE) Engineering Stu-
dents Mentoring Fellowship by INAE Fellows in
academic tenure 2022–2023.
Danfeng Hong (Senior Member, IEEE) received
the M.Sc. degree (summa cum laude) in computer
vision from the College of Information Engineering,
Qingdao University, Qingdao, China, in 2015, and
the Dr.-Ing. degree (summa cum laude) from the
Signal Processing in Earth Observation (SiPEO),
Technical University of Munich (TUM), Munich,
Germany, in 2019.
He is currently a Professor with the Key Labora-
tory of Computational Optical Imaging Technology,
Aerospace Information Research Institute, Chinese
Academy of Sciences (CAS), Beijing, China. Before joining CAS, he has
been a Research Scientist and led a Spectral Vision Working Group at
the Remote Sensing Technology Institute (IMF), German Aerospace Center
(DLR), Oberpfaffenhofen, Germany. He was also an Adjunct Scientist at
the GIPSA-Lab, Grenoble INP, CNRS, Université Grenoble Alpes, Grenoble,
France. His research interests include signal/image processing, hyperspectral
remote sensing, machine/deep learning, artificial intelligence, and their appli-
cations in Earth vision.
Dr. Hong was a recipient of the Best Reviewer Award of the IEEE TGRS
in 2021 and 2022 and the IEEE JSTARS in 2022, the Jose Bioucas Dias
Award for recognizing the Outstanding Paper at WHISPERS in 2021, the
Remote Sensing Young Investigator Award in 2022, and the IEEE GRSS
Early Career Award in 2022. He is a Topical Associate Editor of the
IEEE T RANSACTIONS ON G EOSCIENCE AND R EMOTE S ENSING (TGRS),
an Editorial Board Member of Remote Sensing, and an Editorial Advisory
Board Member of ISPRS Journal of Photogrammetry and Remote Sensing.
5530416
Muhammad Ahmad received the M.S. degree in
electronics engineering from International Islamic
University, Islamabad, Pakistan, in 2011, the Ph.D.
degree in computer science and engineering from
Innopolis University, Innopolis, Russia, in 2019,
and the second Ph.D. degree in cyber-physical
systems from the University of Messina, Messina,
Italy, in 2021.
He is currently working with the National
University of Computer and Emerging Sciences
(FAST-NUCES), Chiniot-Faisalabad Campus,
Chiniot, Pakistan. He has also served as an Assistant Professor, a Lecturer,
an Instructor, a Research Fellow, a Research Associate, and a Research
Assistant for a number of international/national universities. He has also
worked with Ericsson (Mobilink Project) as a Radio Access Network (RAN)
Supervisor. He has authored or coauthored over 70 scientific contributions
to international journals, conferences, and books. He is supervising/co-
supervising several graduates (M.S. and Ph.D.). His research interests include
hyperspectral imaging, remote sensing, machine learning, computer vision,
and wearable computing.
Dr. Ahmad served/serving as a lead/guest editor for several special issues
in journals (SCI/E, JCR). He has delivered a number of invited and keynote
talks and reviewed (reviewing) the technology-leading articles for journals.
Antonio Plaza (Fellow, IEEE) received the M.Sc.
and Ph.D. degrees in computer engineering from
the University of Extremadura, Cáceres, Spain, in
1999 and 2002, respectively.
He is currently a Full Professor and the Head of
the Hyperspectral Computing Laboratory, Depart-
ment of Technology of Computers and Communica-
tions, University of Extremadura. He has authored
more than 600 publications in this field, includ-
ing 342 JCR journal articles (249 in IEEE journals),
24 book chapters, and 330 peer-reviewed conference
proceeding papers. He has guest-edited ten special issues on hyperspectral
remote sensing for different journals. His research interests include hyper-
spectral data processing and parallel computing of remote sensing data.
Prof. Plaza is a fellow of the IEEE for contributions to hyperspectral data
processing and parallel computing of Earth observation data and a member of
Academia Europaea, The Academy of Europe. He was a recipient of the recog-
nition of Best Reviewers of the IEEE G EOSCIENCE AND R EMOTE S ENSING
L ETTERS (in 2009) and the IEEE T RANSACTIONS ON G EOSCIENCE AND
R EMOTE S ENSING (in 2010), for which he served as an Associate Editor
(2007–2012). He was also a member of the steering committee of the IEEE
J OURNAL OF S ELECTED T OPICS IN A PPLIED E ARTH O BSERVATIONS AND
R EMOTE S ENSING (JSTARS). He was a recipient of the Best Column Award
of the IEEE Signal Processing Magazine in 2015, the 2013 Best Paper Award
of the JSTARS journal, and the most highly cited paper (2005–2010) in
the Journal of Parallel and Distributed Computing. He received best paper
awards at the IEEE Workshop on Hyperspectral Image and Signal Processing:
Evolution in Remote Sensing, the IEEE International Conference on Space
Technology, and the IEEE Symposium on Signal Processing and Information
Technology. He served as the Director of Education Activities for the
IEEE Geoscience and Remote Sensing Society (GRSS) (2011–2012) and as
Authorized licensed use limited to: Siksha O Anusandhan University. Downloaded on July 15,2023 at 11:28:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
the President of the Spanish Chapter of IEEE GRSS (2012–2016). He is
currently serving as the Chair of the Publications Awards Committee of IEEE
GRSS and the Vice-Chair of the Fellow Evaluations Committee of IEEE
GRSS. He is also an Associate Editor for IEEE ACCESS (receiving the recog-
nition of Outstanding Associate Editor for the journal in 2017) and was also a
member of the Editorial Board of the IEEE Geoscience and Remote Sensing
Newsletter (2011–2012) and the IEEE Geoscience and Remote Sensing Maga-
zine (2013). He has reviewed more than 500 manuscripts for over 50 different
journals. He served as the Editor-in-Chief of the IEEE T RANSACTIONS ON
G EOSCIENCE AND R EMOTE S ENSING journal for five years (2013–2017)
and is currently serving as the Editor-in-Chief of the IEEE J OURNAL ON
M INIATURIZATION FOR A IR AND S PACE S YSTEMS . He has been included in
the 2018, 2019, and 2020 Highly Cited Researchers List (Clarivate Analytics).
More information is available at http://www.umbc.edu/rssipl/people/aplaza.
Jocelyn Chanussot (Fellow, IEEE) received the
M.Sc. degree in electrical engineering from the
Grenoble Institute of Technology (Grenoble INP),
Grenoble, France, in 1995, and the Ph.D. degree
from the Université de Savoie, Annecy, France,
in 1998.
Since 1999, he has been with Grenoble INP,
where he is currently a Professor of signal and
image processing. He has been a Visiting Scholar at
Stanford University, Stanford, CA, USA; the KTH
Royal Institute of Technology, Stockholm, Sweden;
and the National University of Singapore (NUS), Singapore. Since 2013,
he has been an Adjunct Professor with the University of Iceland, Reykjavik,
Iceland. From 2015 to 2017, he was a Visiting Professor with the University
of California at Los Angeles (UCLA), Los Angeles, CA. He holds the AXA
Chair in remote sensing and is an Adjunct Professor with the Aerospace
Information Research Institute, Chinese Academy of Sciences, Beijing, China.
His research interests include image analysis, hyperspectral remote sensing,
data fusion, machine learning, and artificial intelligence.
Dr. Chanussot is the founding President of the IEEE Geoscience and
Remote Sensing French Chapter (2007–2010), which received the 2010 IEEE
GRS-S Chapter Excellence Award. He has received multiple outstanding paper
awards. He was the Vice-President of the IEEE Geoscience and Remote
Sensing Society, in charge of meetings and symposia (2017–2019). He was
the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image
and Signal Processing, Evolution in Remote sensing (WHISPERS). He was
the Chair (2009–2011) and the Co-Chair of the GRS Data Fusion Technical
Committee (2005–2008). He was a member of the Machine Learning for
Signal Processing Technical Committee of the IEEE Signal Processing Society
(2006–2008) and the Program Chair of the IEEE International Workshop on
Machine Learning for Signal Processing (2009). He is an Associate Editor
for the IEEE T RANSACTIONS ON G EOSCIENCE AND R EMOTE S ENSING, the
IEEE T RANSACTIONS ON I MAGE P ROCESSING, and the P ROCEEDINGS OF
THE IEEE. He was the Editor-in-Chief of the IEEE J OURNAL OF S ELECTED
T OPICS IN A PPLIED E ARTH O BSERVATIONS AND R EMOTE S ENSING (2011–
2015). In 2014, he served as a Guest Editor for the IEEE Signal Processing
Magazine. He is a member of the Institut Universitaire de France (2012–2017)
and a Highly Cited Researcher (Clarivate Analytics/Thomson Reuters).