IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL.

60, 2022 5601117

RRSGAN: Reference-Based Super-Resolution for

Remote Sensing Image
Runmin Dong , Lixian Zhang , and Haohuan Fu , Member, IEEE

Abstract— Remote sensing image super-resolution (SR) plays

an important role by supplementing the lack of original high-
resolution (HR) images in the study scenarios of large spatial
areas or long time series. However, due to the lack of imagery
information in low-resolution (LR) images, single-image super-
resolution (SISR) is an inherently ill-posed problem. Especially,
it is difficult to reconstruct the fine textures of HR images at
large upscaling factors (e.g., four times). In this work, based
on Google Earth HR images, we explore the potential of the
reference-based super-resolution (RefSR) method on remote sens-
ing images, utilizing rich texture information from HR reference
(Ref) images to reconstruct the details in LR images. This method
can use existing HR images to help reconstruct the LR images
of long time series or a specific time. We build a reference-based
remote sensing SR data set (RRSSRD). Furthermore, by adopting Fig. 1. Example of the LR image and the corresponding HR image. For
the generative adversarial network (GAN), we propose a novel convenient comparison, the LR image is bicubic upsampled to the size of the
end-to-end reference-based remote sensing GAN (RRSGAN) for HR image. (a) Planet image (3-m resolution). (b) Google Earth image (0.6-m
SR. RRSGAN can extract the Ref features and align them resolution).
to the LR features. Eventually, the texture information in
the Ref features can be transferred to the reconstructed HR [3], building extraction [4], [5], small object detection [6],
images. In contrast to the existing RefSR methods, we propose and so on. However, due to the limitation of the underlying
a gradient-assisted feature alignment method that adopts the
deformable convolutions to align the Ref and LR features and a technology and the high cost of the hardware, the observed
relevance attention module (RAM) to improve the robustness HR images often suffer from incomplete coverage in terms
of the model in different scenarios (e.g., land cover changes of space or time. They cannot meet the growing demand for
and cloud coverage). The experimental results demonstrate that productions and applications [7]. The image super-resolution
RRSGAN is robust and outperforms the state-of-the-art SISR (SR) technology provides a low-cost and effective way to
and RefSR methods in both quantitative evaluation and visual
results, which indicates the great potential of the RefSR method obtain the HR images by reconstructing HR images from
for remote sensing tasks. Our code and data are available at relatively low-resolution (LR) but easily available images [8].
https://github.com/dongrunmin/RRSGAN. In recent years, single-image super-resolution (SISR) has
Index Terms— Deep learning, remote sensing imagery, super- received great attention. There are many SISR-related stud-
resolution (SR). ies [8]–[10], especially deep learning-based methods [7].
However, because the textures of HR images are excessively
I. I NTRODUCTION destructed in the degrading process and are difficult to be
recovered, the convolutional neural network-based (CNN-

H IGH-RESOLUTION (HR) remote sensing images are

crucial in urban planning [1], semantic labeling [2],
Manuscript received July 29, 2020; revised November 16, 2020 and
December 8, 2020; accepted December 16, 2020. Date of publication
January 18, 2021; date of current version December 3, 2021. This work was
supported in part by the fund of Shanghai Municipal Commission of Economy
and Informatization under Grant 2019-RGZN-01015, in part by the National
Key Research and Development Plan of China under Grant 2017YFA0604500,
in part by the National Natural Science Foundation of China under Grant
51761135015 and Grant U1839206, and in part by the Center for High
Performance Computing and System Simulation, Pilot National Laboratory
for Marine Science and Technology (Qingdao). (Corresponding author:
Haohuan Fu.)
Runmin Dong is with the Department of Earth System Science,
Tsinghua University, Beijing 100084, China, and also with Shanghai Sense-
Time Intelligent Technology Company, Ltd., Shanghai 200030, China (e-mail:
drm17@mails.tsinghua.edu.cn).
Lixian Zhang and Haohuan Fu are with the Department of Earth
System Science, Tsinghua University, Beijing 100084, China (e-mail:
zhanglx18@mails.tsinghua.edu.cn; haohuan@tsinghua.edu.cn).
Digital Object Identifier 10.1109/TGRS.2020.3046045
1558-0644 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
based) SISR methods [11]–[13] often result in blurry effects.
To address this issue, the generative adversarial network-based
(GAN-based) SISR methods [14]–[16] are proposed and lead
to breakthroughs on the visual quality. However, GAN-based
SISR methods tend to generate fake textures and even produce
artifacts. Moreover, due to the limited prior knowledge and the
lack of imagery information in a single LR image, it is still a
challenging problem to reconstruct the fine textures of HR at
large upscaling factors (e.g., four times).
The publicly available Google Earth images have signif-
icantly facilitated the usability of HR (up to 0.3 m) images.
Although many HR images are available, they may fail to meet
the demands of applications (e.g., lacking images of long time
series or specific time [17]), in which case the SR technology
is still needed. However, the existing HR Google Earth images
can facilitate new opportunities for image SR tasks. An LR
remote sensing image, in many cases, can be matched to a

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 5601117
corresponding HR image (as shown in Fig. 1). Intuitively,
a Google Earth HR image can provide extra information and
may help reconstruct fine textures in LR images. Therefore,
for the remote sensing SR task, we consider using publicly
available HR images from Google Earth as reference (Ref)
images to help reconstruct the fine texture from LR images.
Specifically, the Ref images can guide the SR process where
the LR and Ref images contain similar contents, leading to
the SR images that have sharp visual quality and preserve the
ground-truth class [18].
In the computer vision field, some studies have
diverted from SISR and explored reference-based SR
(RefSR) [18]–[20]. RefSR introduces additional reference
images to compensate for the lost details in the LR images.
The state-of-the-art methods follow the pattern of combining
image alignment or patch matching and texture synthesis [20].
Existing RefSR studies are mainly based on two different types
of assumptions: 1) the Ref images and LR images are aligned
well or have high content similarity (e.g., the same object
from different viewpoints or video frames) [19], [21], [22] or
2) the Ref images and LR images sometimes are significantly
misaligned or have uncertain content similarity (e.g., from
web image searches) [20], [23], [24]. However, the above
assumptions do not fully match the remote sensing scenario
in this article, mainly due to the following two reasons.
1) In remote sensing tasks, the HR Ref images can be
easily matched to the LR images at the same location
using the latitude and longitude information. Therefore,
the Ref and LR images have a certain content similarity.
However, image alignment is still necessary due to
the different shooting viewpoints, which results in the
different tilting directions of tall buildings, and deviation
in geographic coordinates (usually within several pixels).
The challenges of alignment are the large spectral dif-
ference between different sensors, land cover changes at
different times, shifts in the viewpoint, and illumination
variations.
2) Due to inevitable land cover changes, cloud coverage,
and missing HR Ref images, there are higher require-
ments for the robustness of the model, especially in the
Ref texture transfer process.
To address the above-mentioned issues, we propose a
novel reference-based remote sensing GAN (RRSGAN)
for SR, as shown in Fig. 2. Our approach works in a
“feature extraction–alignment–transfer” fashion, as shown
in Fig. 3. In the feature alignment process, objects with
a larger offset usually come from tall buildings, which
generally have apparent boundaries. Therefore, we propose
a gradient-assisted feature alignment (GAFA) method to
match the Ref features with the LR features for better use
in the subsequent process. In the feature transfer process,
we propose a relevance attention module (RAM) to suppress
irrelevant information and enhance the relevant information
of the Ref features to improve the robustness of the model.
The contributions of our work are summarized as follows.
1) To the best of our knowledge, we are one of the first
to explore RefSR on remote sensing images. Based on
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
the publicly available HR images from Google Earth,
we build an open-source reference-based remote sensing
super-resolution data set (RRSSRD).
2) We propose an end-to-end RefSR approach for remote
sensing images, named RRSGAN. RRSGAN contains
a feature extraction and alignment module, a multi-
level texture transformer, and GAN-based losses in
both image and gradient domains. Considering the mis-
alignment problem between LR and Ref images and
model robustness with different qualities of Ref images,
we propose a GAFA method and an RAM to fur-
ther release the potential of RefSR in remote sensing
scenarios.
3) We demonstrate that our proposed RRSGAN is superior
to both the state-of-the-art SISR methods and existing
RefSR methods on RRSSRD. Our proposed method is
also robust with different qualities of Ref images and
performs well in real-world images. This work proves
the great potential of the RefSR approach in the field of
remote sensing.
The rest of this article is organized as follows. We introduce
the related work in Section II, including the existing SR meth-
ods for remote sensing images, SISR, and the deep learning-
based RefSR methods. In Section III, we give a detailed
description of the proposed approach. Experimental results and
discussion are provided in Section IV, and Section V concludes
our work.
II. R ELATED W ORK
A. SR for Remote Sensing Images
In the field of remote sensing, depending on the number of
input images, two main categories for improving the resolution
of remotely sensed images are SISR and multi-image super-
resolution (MISR) techniques [25]. MISR techniques in
remote sensing aim to reconstruct high spatial frequency
details from multiple LR versions of the same remote sensing
scenes [26]–[30]. However, we can hardly apply MISR
approaches to applications where multiple remotely sensed
images of the same scene are not possible or difficult to
obtain [31]. For such situations, SISR has become the more
feasible SR technology [32].
With the rapid progress made in deep learning, the deep
learning-based SISR methods outperform the traditional
SISR methods and prove to be promising in the domain
of remote sensing SR [31], [33]. The early CNN-based
methods [34]–[36] usually retrained a network designed for
natural images, such as SRCNN and VDSR, with remote
sensing images. Recent progress of SR for remote sensing
images can be summarized in four aspects.
1) Improving the structure of SR network for remote sens-
ing images. For example, Pan et al. [37] proposed a
residual dense backprojection network (RDBPN)-based
SISR method, which can utilize residual learning in
both global and local manners. Jiang et al. [8] improved
the performance regarding the details of remote sensing
images by adding an edge enhancement module to the
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE
GAN architecture. Zhang et al. [38] presented a scene-
adaptive approach using a multiscale attention network
to improve the SR performance on different scenes.
2) Using prior information to help the reconstruction. For
example, Zhang et al. [39] introduced the varying
saliency maps of different areas in remote sensing
images to drive SISR.
3) Designing an approach for the real-world image SR. For
example, Wang et al. used real LR and HR images,
which can be paired or unpaired remote sensing images,
and proposed an unsupervised learning network named
Cycle-CNN for remote sensing image SR.
4) Improving GAN-based SISR methods. For example,
Lei et al. [10] proposed a coupled-discriminated GANs
(CDGANs) for reconstructing remote sensing images
to relieve the problem that the discriminator is easily
confused in the low-frequency regions.
Besides, there are also many studies for promoting the reso-
lution of other types of remote sensing images, such as radar
images [40], [41] and hyperspectral images [42], [43].
B. SISR
The deep learning-based SISR techniques aim to build
a deep learning model that can learn a “universal” prior
from the given training remotely sensed material and then
infer the missing details of the LR images using the trained
model [10]. Dong et al. [44] pioneered SRCNN, which used
three-layer CNN for mapping LR image and HR image pairs.
Subsequently, the performance of SISR has been improved
by applying more advanced deep learning architectures, such
as residual blocks [45], dense blocks [11], and recursive
blocks [46]. For example, SRResNet [14] employed the
ResNet architecture for SR and an enhanced deep SR (EDSR)
network [12] further improved the results by removing the
unnecessary batch normalization modules. Recently, WDSR
[13] used the strategy of wide activation by expanding
features before the rectified linear unit (ReLU) activation.
DBPN [47] introduces backprojection modules to provide an
error feedback mechanism.
The abovementioned methods aim to minimize mean square
error (MSE) or mean absolute error (MAE) between the
SR and HR images, ignoring human perceptions. Therefore,
GANs [48] have emerged as a promising architecture for
reconstructing single frame images into a more photorealis-
tic and perceptual version. Ledig et al. [14] used a GAN
(SRGAN) to reconstruct a photorealistic single image with
HR. Wang et al. [15] proposed ESRGAN to enhance the
visual quality of reconstructed images further. Ma et al. [16]
proposed a structure-preserving SR (SPSR) method with gra-
dient guidance to generate perceptual-pleasant details, which
achieved state-of-the-art visual results.
C. RefSR
RefSR methods introduce additional Ref images with high-
frequency details to reconstruct the missing fine texture of
LR images. The Ref images used in daily scenarios are
generally from different viewpoints [19], video frames [49],
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
5601117
Fig. 2. Overall workflow of the adversarial learning of RRSGAN.
online image searches [20], and so on. Recent RefSR methods
can be roughly classified into two categories: patch matching
and image aligning. Some studies adopted the patch matching
method to find the most relevant Ref feature for each LR
patch [20], [23]. For example, Zheng et al. [23] proposed a
deep learning-based approach for Ref and LR patch matching
and synthesizing. SRNTT [20] adopted the patch matching
method, which used a pretrained VGG network to extract Ref
and LR features, divided VGG features into small patches,
and then calculated and compared the similarity between
each Ref and LR patch to swap similar texture features. The
advantage of patch matching is that it can handle long-distance
dependence, making the model more robust in the case of sig-
nificant misalignment between Ref and LR images. However,
patch matching is usually computationally intensive and time-
consuming. Besides, it cannot accurately deal with nonrigid
image deformation, which often causes blocky artifacts.
Other studies adopted image aligning methods to alleviate
the abovementioned problems [19], [21], [22]. However, they
are based on the assumption that the Ref image and LR
image are well-aligned or have a high content similarity. For
example, CrossNet [19] applied optical flow to align Ref and
LR images at different scales and then used the aligned Ref
feature to perform multiscale feature fusion and synthesis
using U-Net [50]. However, CrossNet is not an end-to-end SR
method. CrossNet first applies an SISR model (e.g., SRResNet
[14]) to upsample the LR image and then performs the
subsequent “encoder–warping–decoder” process. The remote
sensing scenario is more suitable to adopt image aligning,
as the Ref and LR images have a certain content similarity.
However, the performance of such methods depends mainly
on the alignment quality. Therefore, we propose a gradient-
assisted pyramid, cascading and deformable alignment method
to improve the aligning quality. Moreover, we design an end-
to-end RefSR approach, and the proposed RAM can further
improve the performance and robustness of our approach.
III. A PPROACH
In this article, we aim to utilize a reference image (Ref) to
assist in reconstructing an LR image. The proposed RRSGAN
is a GAN-based RefSR method, and the overall workflow is
shown in Fig. 2. RRSGAN consists of a generator and two dis-
criminators. The generator can utilize the Ref images to help
reconstruct the fine textures in LR images. To generate clear
and visually favorable SR results, we use two discriminators
D I and DG in both image and gradient domains. The gradient
 5601117
Fig. 3. Generator of the proposed RRSGAN. The approach is designed in a “feature extraction–alignment–transfer” structure. First, the Ref features are
extracted and aligned to the LR features. In the texture transfer process, we first extract the LR features and then transfer the aligned Ref features in a
multiscale way.
information can help the model focus on neighboring con-
figurations and better infer the local intensity of sharpness.
Therefore, the generator can benefit from the two discrimina-
tors to learn the fine appearance and focus on avoiding the
distortions of geometric details.
The generator of RRSGAN is an end-to-end network,
as shown in Fig. 3. The approach is designed in a “feature
extraction–alignment–transfer" structure. First, we need to
extract the Ref features and align them to the LR features.
Otherwise, the Ref features cannot be used effectively in the
subsequent texture transfer process. Different from the patch-
based method in SRNTT [20] and flow-based method in Cross-
Net [19], we adopt deformable convolutions to align the Ref
and LR features. Also, considering that objects with a larger
offset usually come from tall buildings, we propose a gradient-
assisted alignment method to improve the performance. Then,
we extract the LR features and perform texture transfer from
the Ref features. Following SRNTT [20], we transfer the
aligned Ref features in a multiscale way. Instead of directly
concatenating the LR and Ref features in the texture transfer,
we propose an RAM to improve the robustness of the model.
The RAM can enhance the relevant information to improve
the performance and suppress the irrelevant information of the
Ref features, which is useful for preventing the irrelevant Ref
features from making the results worse than SISR methods.
We will elaborate on the details of the RRSGAN in the
following sections. In Section III-A, we introduce the GAFA
method. In Section III-B, we describe the end-to-end network,
including the LR feature extractor, RAM, texture transfer,
and discriminator. A group of loss functions is introduced in
Section III-C, and the implementation details of the proposed
method are presented in Section III-D.
A. Gradient Assisted Feature Alignment Method
The proposed GAFA contains a feature extraction module
and a feature alignment module, as shown in Fig. 4(a) and (b),
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
respectively. The purpose is to extract the Ref features and
align them to the LR features. As we analyzed in Section I,
the most difficult objects for feature alignment are usually tall
buildings. Intuitively, the gradient information from the object
boundary is useful for alignment. Therefore, in addition to
extracting the image features, we explicitly calculate the image
gradient to assist with feature alignment. Note that although
the feature extractor can implicitly learn to extract useful
features, including the image gradient, the explicit retention
and utilization of the gradient features make the network
concentrate more on the objects that are prone to deviation as
shown in the ablation studies in Section IV-F2. The gradient
map for an image is obtained by computing the difference
between the adjacent pixels. The gradient calculation of a pixel
x = (x, y) in image I is defined as follows:
Ix (x) = I (x + 1, y) − I (x − 1, y)
I y (x) = I (x, y + 1) − I (x, y − 1)
∇ I (x) = (Ix (x), I y (x))
M(I ) =  ∇ I 2 (1)
where M(·) computes the L 2 norm of the gradient at each
location and M(I ) is referred to as the gradient map of image
I . The image gradient ∇ I can be efficiently computed by using
convolution layers with fixed kernels.
To match the size of the LR image to the Ref image,
we first upscale the LR image to obtain LR↑ (↑ denotes the
bilinear upsampling process). Instead of directly using the Ref
image for alignment, we use Ref↓↑ (↓ denotes the bilinear
downsampling process), resampled by first applying bilinear
downsampling and then upsampling to the original Ref image.
Then, LR↑ and Ref↓↑ are used to estimate the offset between
the LR image and the Ref image. Eventually, we use the
offset to align the Ref features. We use Ref↓↑ rather than
the original Ref image for computing the offset because the
different blur degrees of the Ref and LR↑ images increase the
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE
 assisted feature alignment method. The symbol ↑ denotes the bicubic upsampling process, and ↓ denotes the bicubic downsampling process.
Fig. 4. Gradient
The symbol denotes feature concatenating on the channel. (a) Feature extraction module. (b) Feature alignment module.
difficulty of the alignment. In contrast, the blurry Ref↓↑ image
is domain-consistent with LR↑, which helps the alignment.
The effectiveness of the resample strategy of Ref images is
demonstrated in Section IV-F2. As shown in Fig. 4(b), three
pairs of data are used as inputs for the alignment, in which
each pair contains an image and its corresponding gradient
map.
1) Feature Extraction Module: The feature extraction mod-
ule contains two parallel encoders to extract the image and
gradient features simultaneously. Each encoder consists of four
convolution blocks, and each convolution block is composed
of a convolution layer and a Leaky ReLU. The kernel size of
the convolution layers is set to 5. The stride of the last two
convolutional layers is set to 2 to obtain multiscale features.
The number of feature maps in the image encoder and image
gradient encoder is set to 64 and 16. The multiscale features
include both semantic and textual information, contributing to
feature alignment and transfer. Besides, the features extracted
from the two parallel encoders are concatenated at the same
level.
2) Feature Alignment Module: The structure of the feature
alignment module is shown in Fig. 4(b). We use deformable
convolutions [51] to align the Ref and LR features. Deformable
convolution aligns the features more flexibly, compared to the
explicit motion estimation or image warping. Inspired by the
success of deformable convolution for aligning neighboring
frames in the video field [52], we adopt a pyramid structure
to estimate and propagate the offsets and generate aligned Ref
features at multiple levels.
Specifically, we align the features in a coarse-to-fine manner
for the entire image. After we obtain three-level features in
the feature extraction module, the offsets are predicted at each
level by two convolution layers. Then, deformable convolution
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
5601117
can utilize the predicted offsets to align the Ref features.
Besides, the offsets and aligned features at the lth level can
be further used to help predict the offsets and aligned features
at the (l − 1)th level. At level 1, the cascading refinement
[53] is used to improve the performance of the alignment
further. A subsequent deformable alignment is cascaded to
further refine the coarsely aligned Ref features [52]. Note that
the alignment module does not require explicit supervision.
Also, the offset is jointly learned with the whole model instead
of being trained separately. The effectiveness of the feature
alignment module is evaluated in Section IV-F2.
B. End-to-End Network Structure
Without the requirement for a separate SISR network as in
CrossNet [19] or a separate feature alignment network as in
SRNTT [20], the proposed RRSGAN is trained in an end-
to-end manner. We use the GAN architecture. The generator
includes a feature alignment module (elaborated in Section III-
A), an LR feature extractor, and a texture transformer, which
contains the RAMs. We use two discriminators D I and DG
for the image and gradient domains, respectively.
1) LR Feature Extractor: The input LR image is first
forwarded to the feature extractor, which contains N residual
blocks. Each block consists of two convolution layers and a
Leaky ReLU, as shown in the pink box in Fig. 3. In this work,
the kernel size of the convolution layers is set to 3 × 3, and
the number of feature maps in each layer is set to 64.
2) RAM : Instead of directly combining the Ref and LR
features for texture transfer, we use the correlation between
the Ref and LR features to modify the Ref features before
combining. Intuitively, the relevant information between Ref
and LR features should be enhanced, and the less relevant
information should be suppressed in the Ref features and
 5601117
Fig. 5. Relevance attention module (RAM).
replaced with the LR features. Therefore, as shown in Fig. 5,
we first combine the Ref and LR features and obtain atten-
tion maps through two convolution layers. The normalized
attention maps
 are elementwise multiplied by the Ref features
(denoted as ). Then, we obtain additional information from
the combined features using two other convolution layers,
which are elementwise added to the Ref features (denoted as
+). Finally, the optimized Ref features are
combined with the
LR features on the channel (denoted as ) for the further
texture transfer process. The RAM is designed to improve the
robustness of the model and can be expressed as follows:
f RAM = Ca ( fRef ⊕ f LR )  fRef + Cb ( f Ref ⊕ fLR )
FRAM = f RAM ⊕ f LR
where fRAM represents the optimized Ref features, fRef and
f LR represent the Ref features and LR features, respectively,
Ca (·) represents the computation containing two convolutional
layers and a sigmoid layer, and Cb (·) represents the computa-
tion containing two convolutional layers.
3) Texture Transformer: Inspired by SRNTT [20], we use
residual blocks and skip connections to build a texture trans-
former. We gradually combine different levels of Ref features
into the network, i.e., from level 3 to level 1. The SR texture
reconstruction benefits from the multiscale texture transfer.
There are three stages of texture transfer, corresponding to
the three levels of features. Each stage contains an RAM and
N residual blocks. Besides, the first two stages include an
upsampling module, which upscales the features by subpixel
convolution. In this process, the texture information in the Ref
image can be effectively transferred to the SR results.
4) Discriminator: Inspired by SPSR [16], we use two
discriminators D I and DG for the image and gradient domains,
respectively, which can help the generator learn the fine
appearance and capture the geometric relationship. The two
discriminators use the same network structure, which shares
a similar model with the VGG-13 [54] network. Specifically,
we replace the max-pooling layers with the stride convolutions
using a 4 × 4 kernel size to enable the network to learn its
spatial downsampling. We also use the leaky-ReLU activation
function to replace the ReLU activation function. Therefore,
the discriminator has ten convolutional layers activated by
the leaky-ReLU function, where the number of convolutional
filters increases by a factor of 2 from 64 to 512 kernels as
in the VGG network. Two fully connected layers follow the
resulting 512 feature maps to provide the final assessment.
C. Loss Function
Commonly used loss functions for SR methods include
reconstruction loss Lrec , adversarial loss Ladv [48], and per-
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
ceptual loss Lper [55]. In addition to the above three loss
terms calculated between I SR and I HR , we adopt the gradient
loss from [16], including the reconstruction loss Lg_rec and
the adversarial loss Lg_adv , both of which are computed for
the gradient maps. The gradient loss can help the network
generate plausible geometric structures. Therefore, we have
two discriminators, D I and DG , optimized by L D I and L DG ,
respectively. The overall loss is defined as
Ltotal = Lrec + αLper + βLadv + γ Lg_rec + δLg_adv . (3)
1) Reconstruction Loss: The reconstruction loss aims to
preserve the spatial structure of the LR image. We use the
loss, which has shown effective sharpening of the SR images
and a better convergence than the MSE. The reconstruction
loss Lrec is defined as
Lrec =  I HR − I SR 1 (4)
where I HR denotes the HR image, I SR denotes the SR image
and I SR = G(I LR ), G(·) denotes the generator, and I LR
(2) denotes the LR image.
2) Adversarial Loss: The adversarial loss encourages the
network to generate clear and visually favorable images [14].
The discriminator D I and the generator G are optimized as
follows:
Ladv = − log(D I (G(I LR )) (5)
L D I = − log(D I (I HR )) − log(1 − D I (I SR )) (6)
where D I is designed for distinguishing a real I HR from a
generated I SR .
3) Perceptual Loss: Perceptual loss can enhance the visual
quality of SR images by constraining the content similarity in
the feature space between I SR and I HR . We use the pretrained
19-layer VGG network to extract the features of I SR and I HR .
The perceptual loss Lper is defined as
Lper =  φi (I HR ) − φi (I SR ) 1 (7)
where φi (·) denotes the i th layer output of the VGG19.
4) Gradient Loss: Gradient loss has been demonstrated to
be useful to retain the geometric structure [16]. The gradient
loss restricts the second-order relationship of neighboring
pixels. We obtain the gradients of I SR and I HR by computing
the difference between the adjacent pixels, as shown in (1).
The gradient loss includes the gradient-based reconstruction
loss Lg_rec and the gradient-based adversarial loss Lg_adv ,
which are defined as follows:
Lg_rec =  M(I HR ) − M(I SR ) 1 (8)
Lg_adv = − log(DG (G(M(I LR ))) (9)
L DG = − log(DG (M(I HR ))) − log(1 − DG (M(I SR )))
(10)
where M(·) denotes the gradient operation and DG is designed
for distinguishing a real M(I HR ) from a generated M(I SR ).
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE
Fig. 6. Examples of HR-Ref pairs in the RRSSRD data set.
TABLE I
I NFORMATION A BOUT THE RRSSRD
D. Implementation Details
Following the standard protocol, we obtain the LR images
during training by downsampling the HR images using a
bicubic kernel with downsampling factor r = 4. For each
input minibatch, we randomly crop 16 64 × 64 patches from
LR images. The corresponding HR patches have a size of
256 × 256. The texture transformer in an SR network contains
three stages, each consists of 16 residual blocks. For the
discriminators, we adopt a VGG-style network without BN
layers. We set the weight hyperparameters for α, β, γ , and δ
to 0.1, 0.001, 1, and 0.001, respectively. The Adam optimizer
is used for optimization with the parameters of β1 = 0.9,
β2 = 0.999, and  = 1 × 10−8 . The learning rates for both
the generator and discriminators are set to 1 × 10−4 and are
reduced to half with 50k, 100k, and 200k iterations. We first
warm up the network for 30k iterations where only Lrec and
Lg_rec are applied. Then, we use all losses to train a total
of 300k iterations. We implement our models with the PyTorch
framework and train them using 16 NVIDIA GTX 1080Ti
GPUs.
IV. E XPERIMENTS
A. Data Sets
To the best of our knowledge, the existing common data
sets used for SR of remote sensing tasks [56], [57] do
not provide coordinate information for each image, limit-
ing the matching of reference images. Therefore, we build
a benchmark data set for RefSR technology in this work,
named the RRSSRD. This data set covers common classes
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
5601117
of remote sensing scenes, including airport, bare land, beach,
bridge, center, commercial, dense-residential, farmland, for-
est, industrial, meadow, medium-residential, park, parking,
playground, pond, port, river, sparse-residential, viaduct,
and so on.
Information about the RRSSRD is shown in Table I. Exam-
ples of the HR-Ref pair in RRSSRD are shown in Fig. 6.
RRSSRD consists of 4047 pairs of HR-Ref images with RGB
bands. The HR images are acquired from WorldView-2 and
GaoFen-2, and depict Xiamen and Jinan City, China. The Ref
images are collected from Google Earth in 2019 with a spatial
resolution of 0.6 m. We downsample each HR image 4 times
to a LR image. The HR and Ref images are sized 480 × 480
pixels, and correspondingly, the LR images are sized 120×120
pixels.
Considering the model performance on different image
sources and locations, we build four test data sets. Each test
set consists of 40 pairs of HR-Ref images. In the first test set,
the images are collected from WorldView-2 and depict Xiamen
City, China. The images in the second test set are also taken in
Xiamen City, but the HR images are collected from Microsoft
Virtual Earth in 2018 with a spatial resolution of 0.5 m. In the
third test set, the HR images are acquired from the GaoFen-
2 (GF-2) satellite in 2018 with a spatial resolution of 0.8 m
and depict Jinan City, China. The HR images in the fourth test
set are collected from Microsoft Virtual Earth and depict Jinan
in 2018 with a spatial resolution of 0.5 m. Note that all the Ref
images are collected from Google Earth in 2019 with a spatial
resolution of 0.6 m. LR images are obtained by ×4 bicubic
downsampling from the HR images and are sized 120 × 120
 5601117
pixels. The Ref images are resized to 480 × 480 pixels, which
is the same size as the HR images.
Furthermore, we test our method on real-world remotely
sensed images from the GaoFen-1 (GF-1) satellite with a
spatial resolution of 2 m. The corresponding Ref images are
collected from Google Earth and have a spatial resolution
of 0.6 m.
B. Evaluation Metrics
The PSNR and SSIM have been used as standard evaluation
metrics in image SR [58]. Nevertheless, as revealed in some
recent studies [59], [60], super-resolved images may some-
times have high PSNR and SSIM scores with over smoothed
results but tend to lack realistic visual results. At this moment,
apart from the PSNR, the perception index (PI) [59] and the
learned perceptual image patch similarity (LPIPS) [60] are
included in our experiments. Besides, the PI and natural image
quality evaluator (NIQE) [61] can be used as evaluation met-
rics on real-world images. The NIQE and PI were originally
introduced as nonreference image quality assessment methods
based on low-level statistical features [62]. The NIQE is
obtained by computing the 36 identical natural scene statistical
(NSS) features from patches of the same size from the image
[61]. The PI is calculated by incorporating the criteria of Ma
et al. [63] and NIQE as follows:
1 SSIM because they focus on preserving the spatial structure
PI = ((10 − Ma) + NIQE). (11)
2
The LPIPS is a full-reference metric that measures percep-
tual image similarity using a pretrained deep network. We use
the AlexNet [64] model to compute the l2 distance in the
feature space. LPIPS can be calculated using a given image y
and a ground-truth image y0 as follows:
 1   
LPIPS(y, y0 ) = wl  f l − f l 2 (12)
Hl Wl h,w h,w 0h,w 2 assumptions in common scenarios, and it is unreasonable to
l
where Hl and Wl represent the height and width of the lth
l
layer, respectively, f h,w and f0lh,w represent the features of the
corresponding y and y0 of the lth layer at location (h, w),
respectively, wl is a learned weight vector, and  is the
elementwise multiplication operation. Note that, in contrast
to PSNR and SSIM, lower PI and LPIPS indicate better SR
results.
C. Quantitative and Qualitative Comparison With Different
Methods
In this section, we compare our proposed method with
state-of-the-art SISR and RefSR methods on the four test
data sets. The compared SISR methods include five CNN-
based SISR methods (i.e., VDSR [11], SRResNet [14], MDSR
[12], WDSR [13], and DBPN [47]), two state-of-the-art SR
methods for remote sensing images (i.e., RDBPN [37] and
Cycle-CNN [65]), and two GAN-based SISR methods (i.e.,
ESRGAN [15] and SPSR [16]). Two RefSR methods, i.e., the
recently proposed CrossNet [19] and SRNTT [20], are also
included in the comparison. Note that in these two methods,
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
CrossNet is a CNN-based method and SRNTT is a GAN-
based method. All these methods are fully optimized on
our training data set to obtain their best performance for a
fair comparison. Note that Cycle-CNN aims to reconstruct
the real-world images and requires real LR images that are
not generated from HR images. Thus, we add the real LR
images from the GF-1 satellite for Cycle-CNN method and
also compare our method with the Cycle-CNN both on the
four test data sets and the real-world GF-1 data. The results
using bicubic interpolation (Bicubic) are also included for
comparison.
For better comparison with both GAN-based methods and
CNN-based methods, we train two networks, i.e., RRSGAN
and RRSNet. RRSNet uses only the reconstruction loss, a sim-
plified version of RRSGAN, with the discriminators removed.
RRSNet is evaluated to make a fair comparison with the CNN-
based methods.
We quantitatively evaluated the SR results using four met-
rics, including PI, LPIPS, PSNR, and SSIM. In each row,
the best result is highlighted in red. As shown in Table II,
RRSNet exhibits the highest scores in the metric of PSNR and
SSIM, whereas RRSGAN achieves the best performance in the
metric of LPIPS on all four test data sets. For the PI metric,
our proposed approach outperforms the other SR methods on
most test data sets.
Generally, CNN-based methods have better PSNR and
of the LR images. However, the SR results of CNN-based
methods suffer from a lack of realistic visual appearance,
which causes worse LPIPS and PI. In contrast, GAN-based
methods obtain better LPIPS and PI, as they use adversarial
loss and perceptual loss, encouraging the network to generate
visually favorable results. Besides, we notice that the per-
formances of CrossNet and SRNTT are not satisfactory. The
reason may be that the models are designed based on specific
apply them to remote sensing scenarios directly. Owning to
the usage of Ref features, RRSNet surpasses other CNN-
based methods by a large margin. Compared with GAN-
based methods, RRSGAN is not only very competitive on the
image quality assessment but also performs well on PSNR and
SSIM. The reason is that RRSGAN can utilize rich texture
information from Ref images to reconstruct the details in LR
images.
A visual comparison is presented in Fig. 7 and can fur-
ther explain the quantitative results. The results of bicubic
interpolation cannot produce extra details. Owning to the
learning-based technologies, CNN-based SISR methods, such
as SRResNet, MDSR, WDSR, DBPN, RDBPN, and Cycle-
CNN, can reconstruct some texture details but still suffer
from blurry contours due to the simplex optimization objective
function. GAN-based SISR methods, such as ESRGAN and
SPSR, have a better visual appearance but generate artificial
artifacts and worsen the reconstruction results. The SR results
of SRNTT suffer from the problem of blocky artifacts due to
its patch matching method. Compared with other SR methods,
our proposed approach recovers finer texture details, and the
results are more natural and realistic.
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE 5601117

Fig. 7. Visual comparison of our methods with different SR methods on the test sets.

D. Robustness of Our Proposed Method we simulate four scenarios, including Ref images from dif-
In practical applications, RefSR methods need to be suf- ferent image sources, covered by clouds, mismatched, and
ficiently robust against various Ref images with different missing. Correspondingly, the four kinds of Ref images
quality levels. To test the robustness of our proposed approach, are retrieved from Microsoft Virtual Earth (MS), covered

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 5601117 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

TABLE II
Q UANTITATIVE C OMPARISON W ITH D IFFERENT M ETHODS . VDSR, SRR ES N ET, MDSR, WDSR, DBPN, RDBPN, AND C YCLE -CNN A RE CNN-BASED
SISR M ETHODS . ESRGAN AND SPSR A RE GAN-BASED SISR M ETHODS . C ROSS N ET AND SRNTT A RE R EF SR M ETHODS . F OR PSNR AND
SSIM, A H IGHER S CORE I NDICATES B ETTER , W HEREAS FOR PI AND LPIPS, A L OWER S CORE I NDICATES B ETTER . I N E ACH ROW,
THE B EST R ESULT I S H IGHLIGHTED IN Red

TABLE III
R ESULTS OF THE U SE OF D IFFERENT R EF I MAGES ON THE F IRST T EST S ET. F OR PSNR AND SSIM, A H IGHER S CORE I NDICATES B ETTER , W HEREAS
FOR LPIPS, A L OWER S CORE I NDICATES B ETTER . Red I NDICATES THE B EST AND Blue I NDICATES THE S ECOND -B EST R ESULTS

TABLE IV
R ESULTS OF THE A BLATION S TUDY ON R EF T EXTURE T RANSFER . F OR PSNR AND SSIM, A H IGHER S CORE I NDICATES B ETTER , W HEREAS FOR PI AND
LPIPS, A L OWER S CORE I NDICATES B ETTER . I N E ACH C OLUMN , THE B EST R ESULT I S H IGHLIGHTED IN R ED

by clouds (Cloud), irrelevant images (Irrelevant), and black
images (Black). To further demonstrate the robustness and
effectiveness of our proposed method, we also use the bicubic
upsampled LR images (LR X4) or HR images (HR) as
references.
We calculate the quantitative results of the reconstruction
results under the conditions mentioned above on the first
test set. The results presented in Table III show that the
proposed approach is robust in handling the most com-
mon distortion cases in remote sensing. Even when the
“Black” images or “Cloud” images are used as the references,
the results of our method are still better than those of the
GAN-based SISR methods (compared with Table II). This is
due to the multiscale reconstruction structure and the gradient
loss, which guarantees the baseline performance, and the
RAM, which suppresses the irrelevant information in the

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE 5601117

Fig. 8. SR results of using different Ref images. The first and the third rows represent the different Ref images. The second and the fourth rows represent
the corresponding SR results.

Ref features. Although the SR results of the model with
irrelevant Ref images have similar qualitative scores in terms
of PSNR and SSIM to that of the model with Google image,
a better relevance can produce more realistic textures, which
is reflected in the metric of LPIPS. As shown in Fig. 8, using
relevant Ref images, the SR results show sharp edges and
display clear image content. In contrast, using irrelevant Ref
images, the model degenerates to an SISR method and cannot
reconstruct more details.
Specifically, the model achieves the best performance when
we use the HR images as references. It demonstrates the
effectiveness of the texture transfer from the Ref images,

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 5601117 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

Fig. 9. Experimental results in a real-world scenario. The results of the PI and NIQE of each SR image are presented. A lower score indicates better results.

and the performance can be improved with highly relevant TABLE V

Ref images. R ESULTS OF THE A BLATION S TUDY ON THE F EATURE A LIGNMENT
M ETHOD . F OR PSNR AND SSIM, A H IGHER S CORE I NDICATES B ET-
TER , W HEREAS FOR PI AND LPIPS, A L OWER S CORE I NDICATES
E. Test on Real-World Data B ETTER . R ED I NDICATES THE B EST R ESULTS

In this section, we show the SR results of real-world

GaoFen-1 satellite data. As there are no HR images, we use
two nonreference image quality metrics, i.e., PI and NIQE,
to evaluate the perception results. For both metrics, a lower
score indicates a better reconstruction result. As presented
in Fig. 9, the reconstruction results in a real-world scenario
reveal that our proposed approach achieves the best perception
results than those of the other SISR and Ref methods. Note
that Cycle-CNN is an SR method toward real-world remote
sensing images, and we add the real LR images from the
GF-1 satellite for this method. Although our method does not
use the GF-1 satellite images in the training stage, the result
of RRSNet is still competitive compared with Cycle-CNN.
It demonstrates that our method performs well in the real-
world scenario, owing to the use of real Ref images.

F. Ablation Studies
In this section, we verify the effectiveness of each com-
ponent of our approach, including the Ref textual transfer,
GAFA method, RAM, and gradient loss. We also discuss the
hyperparameter tuning of loss weight and model efficiency.
1) Effectiveness of Ref Textual Transfer: To verify the
effectiveness of the Ref textual transfer, we conduct ablation
experiments. We use the same training strategy and network
parameters as introduced in Section III-C, except for the used
level of Ref features in texture transfer. Note that we keep
the same feature alignment module and the same number of
residual blocks at each stage in the comparison experiments
instead of removing each entire stage of texture transfer to
avoid different performances caused by the different depths
of the network. “Without texture transfer” means that we do
not use Ref features in the SR process, and this method can
be regarded as an SISR method. Then, we gradually add
different levels of Ref features into the network from level 1 to
level 3. “With 1-level texture transfer” means that we only
use the Ref level-1 features, which is shown as the largest
orange box in Fig. 3. “With 2-level texture transfer” means
that we use the Ref level 1 and Ref level 2 features. “With
3-level texture transfer” means that we use all three-level Ref
features. As shown in Table IV, the use of Ref features can
significantly improve the performance of SR results compared
with the SISR method. Gradually combining more in-depth
Ref features can improve the performance of SR results.
Therefore, we use three levels of Ref features in our method.
2) Effectiveness of the GAFA Method: We discuss the effect
of feature alignment between Ref and LR images. We conduct

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE 5601117

TABLE VI
R ESULTS OF THE A BLATION S TUDY ON F EATURE A LIGNMENT M ETHOD IN T ERMS OF THE N UMBER OF A LIGNMENT L EVEL . F OR PSNR AND SSIM,
A H IGHER S CORE I NDICATES B ETTER , W HEREAS FOR PI AND LPIPS, A L OWER S CORE I NDICATES B ETTER . I N E ACH C OLUMN , THE B EST
R ESULT I S H IGHLIGHTED IN R ED

TABLE VII TABLE VIII

R ESULTS OF THE A BLATION S TUDY ON F EATURE A LIGNMENT M ETHOD IN
T ERMS OF THE R ESAMPLING S TRATEGY OF R EF I MAGES . W E R EPORT
LPIPS R ESULTS ON T EST S ETS . A L OWER S CORE I NDICATES B ET-
TER . I N E ACH C OLUMN , THE B EST R ESULT I S H IGHLIGHTED IN
R ED
R ESULTS OF THE A BLATION S TUDY ON RAM. W E R EPORT LPIPS
R ESULTS ON T EST S ETS . A L OWER S CORE I NDICATES B ETTER . T HE
B EST R ESULT I S H IGHLIGHTED IN R ED

with “Baseline.” “DConv” can further improve the perfor-

Fig. 10. Attention masks of different levels in RAM.
four comparison experiments, as shown in Table V. “Baseline”
means that the extracted Ref features are directly used without
the process of aligning to the LR features. “FlowNet” implies
that the Ref features are aligned to the LR features by the flow-
based method, which is applied in CrossNet [66]. “DConv
without Grad” and “DConv with Grad” align the Ref features
to the LR features by the deformable convolution method. The
only difference is whether the gradient features are extracted
and used to assist feature alignment.
As shown in Table V, “Baseline” only performs better than
SRGAN (shown in Table II), which indicates that the Ref
features cannot be reasonably used without feature alignment.
“FlowNet” can effectively improve the performance compared
mance compared with the flow-based method. The gradient-
assisted method can effectively improve the performance in
PSNR, SSIM, and LPIPS, which indicates that the recon-
struction results are more credible and realistic than those not
using the gradient-assisted method. The experimental results
prove that feature alignment is an essential part of the RefSR
approach and that a better feature alignment method can obtain
better SR results.
We conduct the ablation study on the feature alignment
method in terms of the number of alignment levels. As we
discuss the optimal level of the Ref features in textual transfer
in Section IV-F1, the number of alignment levels requires
at least three to provide different Ref features. Therefore,
we compare the performance of three-level alignment with
four-level alignment. As shown in Table VI, the results of
the three-level alignment method are similar to the results of
the four-level alignment method, indicating that the features
of level 4 have a minor contribution to improving feature
alignment. Considering the computational cost, we use the
three-level offset to align the LR domain and the Ref domain
in our method.
We also conduct the ablation study on the feature alignment
method in terms of the resampling strategy of Ref images.
As shown in Table VII, the feature alignment method is
effective even without the resampling strategy, while the
resampling strategy further improves the results. The reason
is that the resampled Ref features match the frequency band
of the upsampled LR features, which can reduce the difficulty
of feature alignment.
3) Effectiveness of RAM : To verify the effectiveness of the
RAM, we remove the RAM and directly combine the Ref and
LR features in the feature transfer process. We compare the
results by using different Ref images. As shown in Table VIII,

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 5601117 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022

TABLE IX
R ESULTS OF THE A BLATION S TUDY ON G RADIENT L OSS . F OR PSNR AND SSIM, A H IGHER S CORE I NDICATES B ETTER W HEREAS FOR PI AND LPIPS,
A L OWER S CORE I NDICATES B ETTER . I N E ACH C OLUMN , THE B EST R ESULT I S H IGHLIGHTED IN R ED

TABLE X
R ESULTS OF D IFFERENT L OSS W EIGHTS . F OR PSNR AND SSIM, A H IGHER S CORE I NDICATES B ETTER , W HEREAS FOR PI AND LPIPS, A L OWER S CORE
I NDICATES B ETTER . I N E ACH C OLUMN , THE B EST R ESULT I S H IGHLIGHTED IN R ED

TABLE XI
C OMPARISON OF M ODEL PARAMETERS AND I NFERENCE RUNTIME . VDSR, SRR ES N ET, MDSR, WDSR, DBPN, RDBPN, AND C YCLE -CNN A RE
CNN-BASED SISR M ETHODS . ESRGAN AND SPSR A RE GAN-BASED SISR M ETHODS . C ROSS N ET AND RRSN ET A RE CNN-BASED R EF SR
M ETHODS . SRNTT AND RRSGAN A RE GAN-BASED R EF SR M ETHODS

the use of RAM can effectively improve the robustness of
the model in different scenarios. The reason is that RAM can
suppress the influence of the less relevant information in the
Ref features. The attention masks of different levels in RAM
are presented in Fig. 10 and can further explain the process.
As shown by the dark areas in the red rectangles, the areas
with land cover changes between the LR image and the Ref
image, caused by different seasons or building changes, have
received less attention. The attention is focused on the relevant
area between the LR image and the Ref image, as shown by
the bright area in the green rectangles. Therefore, the RAM
can improve the robustness of the model by suppressing the
irrelevant information and enhancing the relevant information
between the LR features and the Ref features.
4) Effectiveness of the Gradient Loss: We analyze the effect
of the gradient loss. “Baseline” means that we only use
the common SR loss functions, including the reconstruction
loss Lrec , the adversarial loss Ladv , and the perceptual loss
Lper . The gradient-based reconstruction loss Lg_rec and the
gradient-based adversarial loss Lg_adv are added sequentially.
As shown in Table IX, gradient loss can improve the PI and
LPIPS compared with those of the baseline model. It indicates
that the use of the gradient loss can yield a more realistic
visual appearance. Besides, although the additional gradient
discriminator increases the training time and training difficulty,
it does not increase the inference time of the test phase.
5) Hyperparameter Tuning of Loss Weight: We perform
ablation experiments to understand the impact of different
loss terms in (1). We use the same training strategy and
network parameters as introduced in Section III-C, except
for different loss weights. First, we experiment the effect of
using only reconstruction loss Lrec , i.e., with RRSNet, when
α, β, γ , and δ are all set to 0. As shown in Table X,
the highest PSRN and SSIM values can be obtained using only
reconstruction loss compared with other loss weight settings.
However, the reconstruction loss function often leads to overly
smoothed results and is weak in restoring natural and realistic
textures. The introduction of the gradient loss Lgrec and Lgadv
can greatly improve the visual quality of reconstruction, which
has been verified in Section IV-F4.
To determine the appropriate setting of the loss weights,
based on the commonly used loss weights in the SR methods
[15], [16], we experiment with three sets of hyperparame-
ters. Following the setting of different loss weights in [16],
gradient-based loss weight settings are consistent with image-
based loss weight settings, i.e., γ is set to 1 and δ is equal

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE
to β. From Table X, we can see that the reasonable use of
the adversarial loss Ladv and the perceptual loss Lper can
greatly improve the perceptual effect and contribute to better
PI and LPIPS values. However, excessive adversarial loss and
perceptual loss weights can reduce the performance of the SR
results or even lead to the failure of the feature alignment
module. The reason is that our model learns to align feature
maps in an unsupervised fashion, where we do not explicitly
define a loss term for pixelwise offset estimation. It means that
the feature alignment indirectly benefits from the final super-
vision of the SR results. In such a situation, the reconstruction
loss can provide clearer guidance of learning feature alignment
than the adversarial loss and the perceptual loss. To balance
the image reconstruction effect and perceptual effect, we set
the weight hyperparameters for α, β, γ , and δ to 0.1, 0.001,
1, and 0.001, respectively.
6) Model Efficiency: In Table XI, we report the number
of model parameters, computational complexity, the training
time, and the inference time (in the GPU mode) of different
SISR and RefSR methods. For the inference time, all the
approaches are run on an NVIDIA GTX 1080Ti GPU and
tested on 120 × 120 LR images. Correspondingly, the Ref
input images are 480 ×480 pixels and are only used in the Ref
methods. For the training time, some of the SISR methods are
trained for 1 000 000 iterations, including VDSR, SRResNet,
EDSR, MDSR, WDSR. Cycle-CNN, and CrossNet are trained
for 500 000 iterations. SRNTT is trained for 400 000 iterations.
The rest of the methods are trained for 250 000 iterations. Note
that the training time of SRNTT is measured only for the
network training phase, excluding the offline feature swapping
phase. In general, the training time of GAN-based methods is
longer than the CNN-based methods. The inference time of the
Ref methods is longer than that of the SISR methods due to the
extra processing of the Ref images. Compared with the results
of SRNTT, our proposed methods can effectively reduce the
inference time. In future work, we will further optimize our
approach in terms of model efficiency.
V. C ONCLUSION
In this article, we explore the use of reference (Ref) images
to assist in the reconstruction of LR images in remote sensing
tasks. We build a benchmark data set and propose RRSGAN,
an end-to-end network with a GAFA module and a texture
transformer. GAFA extracts the Ref features and aligns them
to LR features. A texture transformer can effectively utilize
the aligned Ref features to help reconstruct the fine textures in
LR images. Experimental results demonstrate the effectiveness
and robustness of RRSGAN. This work also proves the great
potential of the RefSR approach in the field of remote sensing.
In future work, we will further explore the performance of
RefSR at a larger upscaling factor (e.g., eight times) and
optimize our approach in terms of model efficiency.
R EFERENCES
[1] R. Mathieu, C. Freeman, and J. Aryal, “Mapping private gardens
in urban areas using object-oriented techniques and very high-
resolution satellite imagery,” Landscape Urban Planning, vol. 81, no. 3,
pp. 179–192, Jun. 2007.
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
5601117
[2] B. Pan, Z. Shi, X. Xu, T. Shi, N. Zhang, and X. Zhu, “CoinNet: Copy
initialization network for multispectral imagery semantic segmentation,”
IEEE Geosci. Remote Sens. Lett., vol. 16, no. 5, pp. 816–820, May 2019.
[3] R. Dong, W. Li, H. Fu, M. Xia, J. Zheng, and L. Yu, “Semantic
segmentation based large-scale oil palm plantation detection using
high-resolution satellite images,” Proc. SPIE, vol. 10988, May 2019,
Art. no. 109880D.
[4] W. Li, C. He, J. Fang, J. Zheng, H. Fu, and L. Yu, “Seman-
tic segmentation-based building footprint extraction using very high-
resolution satellite images and multi-source GIS data,” Remote Sens.,
vol. 11, no. 4, p. 403, Feb. 2019.
[5] S. Yuan et al., “Long time-series analysis of urban development based
on effective building extraction,” Proc. SPIE, vol. 11398, Apr. 2020,
Art. no. 113980M.
[6] Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “SOD-MTGAN: Small
object detection via multi-task generative adversarial network,” in Proc.
Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 206–221.
[7] W. Yang, X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, “Deep
learning for single image super-resolution: A brief review,” IEEE Trans.
Multimedia, vol. 21, no. 12, pp. 3106–3121, Dec. 2019.
[8] K. Jiang, Z. Wang, P. Yi, G. Wang, T. Lu, and J. Jiang, “Edge-enhanced
GAN for remote sensing image superresolution,” IEEE Trans. Geosci.
Remote Sens., vol. 57, no. 8, pp. 5799–5812, Aug. 2019.
[9] N. Huang, Y. Yang, J. Liu, X. Gu, and H. Cai, “Single-image super-
resolution for remote sensing data using deep residual-learning neural
network,” in Proc. Int. Conf. Neural Inf. Process. Guangzhou, China:
Springer, 2017, pp. 622–630.
[10] S. Lei, Z. Shi, and Z. Zou, “Coupled adversarial training for remote
sensing image super-resolution,” IEEE Trans. Geosci. Remote Sens.,
vol. 58, no. 5, pp. 3633–3643, May 2020.
[11] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution
using very deep convolutional networks,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 1646–1654.
[12] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual
networks for single image super-resolution,” 2017, arXiv:1707.02921.
[Online]. Available: http://arxiv.org/abs/1707.02921
[13] J. Yu et al., “Wide activation for efficient and accurate image super-
resolution,” 2018, arXiv:1808.08718. [Online]. Available: http://arxiv.
org/abs/1808.08718
[14] C. Ledig et al., “Photo-realistic single image super-resolution using
a generative adversarial network,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jul. 2017, pp. 4681–4690.
[15] X. Wang et al., “ESRGAN: Enhanced super-resolution generative adver-
sarial networks,” in Proc. Eur. Conf. Comput. Vis. Workshops (ECCVW),
Sep. 2018.
[16] C. Ma, Y. Rao, Y. Cheng, C. Chen, J. Lu, and J. Zhou,
“Structure-preserving super resolution with gradient guidance,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020,
pp. 7769–7778.
[17] Q. Liu, J. C. Trinder, and I. L. Turner, “Automatic super-resolution
shoreline change monitoring using Landsat archival data: A case study
at Narrabeen–Collaroy Beach, Australia,” Proc. SPIE, vol. 11, no. 1,
Mar. 2017, Art. no. 016036.
[18] Z.-S. Liu, W.-C. Siu, and Y.-L. Chan, “Reference based face super-
resolution,” IEEE Access, vol. 7, pp. 129112–129126, 2019.
[19] H. Zheng, M. Ji, H. Wang, Y. Liu, and L. Fang, “Crossnet: An end-to-end
reference-based super resolution network using cross-scale warping,” in
Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 88–104.
[20] Z. Zhang, Z. Wang, Z. Lin, and H. Qi, “Image super-resolution by
neural texture transfer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2019, pp. 7982–7991.
[21] H. Yue, X. Sun, J. Yang, and F. Wu, “Landmark image super-resolution
by retrieving Web images,” IEEE Trans. Image Process., vol. 22, no. 12,
pp. 4865–4878, Dec. 2013.
[22] Y. Wang, Y. Liu, W. Heidrich, and Q. Dai, “The light field attachment:
Turning a DSLR into a light field camera using a low budget camera
ring,” IEEE Trans. Vis. Comput. Graph., vol. 23, no. 10, pp. 2357–2364,
Oct. 2017.
[23] H. Zheng et al., “Learning cross-scale correspondence and patch-
based synthesis for reference-based super-resolution,” in Proc. Brit.
Mach. Vis. Conf. (BMVC), T.-K. Kim, S. Zafeiriou, G. Brostow, and
K. Mikolajczyk, Eds. BMVA Press, Sep. 2017, pp. 138.1–138.13, doi:
10.5244/C.31.138.
[24] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo, “Learning texture trans-
former network for image super-resolution,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2020, pp. 5791–5800.
 5601117
[25] L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, and L. Zhang, “Image super-
resolution: The techniques, applications, and future,” Signal Process.,
vol. 128, pp. 389–408, Nov. 2016.
[26] R. Y. Tsai and T. S. Huang, “Multiframe image restoration and regis-
tration,” Adv. Comput. Vis. Image Process., vol. 1, no. 2, pp. 317–339,
1984.
[27] T. Akgun, Y. Altunbasak, and R. M. Mersereau, “Super-resolution
reconstruction of hyperspectral images,” IEEE Trans. Image Process.,
vol. 14, no. 11, pp. 1860–1875, Nov. 2005.
[28] J. Ma, J. C.-W. Chan, and F. Canters, “An operational superresolution
approach for multi-temporal and multi-angle remotely sensed imagery,”
IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 5, no. 1,
pp. 110–124, Feb. 2012.
[29] H. Shen, M. K. Ng, P. Li, and L. Zhang, “Super-resolution reconstruction
algorithm to MODIS remote sensing images,” Comput. J., vol. 52, no. 1,
pp. 90–100, Feb. 2008.
[30] F. Li, X. Jia, D. Fraser, and A. Lambert, “Super resolution for remote
sensing images based on a universal hidden Markov tree model,” IEEE
Trans. Geosci. Remote Sens., vol. 48, no. 3, pp. 1270–1278, Mar. 2010.
[31] R. Fernandez-Beltran, P. Latorre-Carmona, and F. Pla, “Single-frame
super-resolution in remote sensing: A practical overview,” Int. J. Remote
Sens., vol. 38, no. 1, pp. 314–354, Jan. 2017.
[32] D. Yang, Z. Li, Y. Xia, and Z. Chen, “Remote sensing image super-
resolution: Challenges and approaches,” in Proc. IEEE Int. Conf. Digit.
Signal Process. (DSP), Jul. 2015, pp. 196–200.
[33] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep
residual networks for single image super-resolution,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jul. 2017,
pp. 136–144.
[34] Y. Luo, L. Zhou, S. Wang, and Z. Wang, “Video satellite imagery super
resolution via convolutional neural networks,” IEEE Geosci. Remote
Sens. Lett., vol. 14, no. 12, pp. 2398–2402, Dec. 2017.
[35] Z. Shao and J. Cai, “Remote sensing image fusion with deep convolu-
tional neural network,” IEEE J. Sel. Topics Appl. Earth Observ. Remote
Sens., vol. 11, no. 5, pp. 1656–1669, May 2018.
[36] A. Xiao, Z. Wang, L. Wang, and Y. Ren, “Super-resolution for ‘Jilin-1’
satellite video imagery via a convolutional network,” Sensors, vol. 18,
no. 4, p. 1194, Apr. 2018.
[37] Z. Pan, W. Ma, J. Guo, and B. Lei, “Super-resolution of single remote
sensing image based on residual dense backprojection networks,” IEEE
Trans. Geosci. Remote Sens., vol. 57, no. 10, pp. 7918–7933, Oct. 2019.
[38] S. Zhang, Q. Yuan, J. Li, J. Sun, and X. Zhang, “Scene-adaptive remote
sensing image super-resolution using a multiscale attention network,”
IEEE Trans. Geosci. Remote Sens., vol. 58, no. 7, pp. 4764–4779,
Jul. 2020.
[39] L. Zhang, D. Chen, J. Ma, and J. Zhang, “Remote-sensing image
superresolution based on visual saliency analysis and unequal recon-
struction networks,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 6,
pp. 4099–4115, Jun. 2020.
[40] I. Yanovsky, B. H. Lambrigtsen, A. B. Tanner, and L. A. Vese, “Efficient
deconvolution and super-resolution methods in microwave imagery,”
IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 8, no. 9,
pp. 4273–4283, Sep. 2015.
[41] S. Kanakaraj, M. S. Nair, and S. Kalady, “SAR image super resolution
using importance sampling unscented Kalman filter,” IEEE J. Sel.
Topics Appl. Earth Observ. Remote Sens., vol. 11, no. 2, pp. 562–571,
Feb. 2018.
[42] X. Xu et al., “A new spectral-spatial sub-pixel mapping model for
remotely sensed hyperspectral imagery,” IEEE Trans. Geosci. Remote
Sens., vol. 56, no. 11, pp. 6763–6778, Nov. 2018.
[43] C. Yi, Y.-Q. Zhao, and J. C.-W. Chan, “Hyperspectral image super-
resolution based on spatial and spectral correlation fusion,” IEEE Trans.
Geosci. Remote Sens., vol. 56, no. 7, pp. 4165–4177, Jul. 2018.
[44] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional
network for image super-resolution,” in Proc. Eur. Conf. Comput. Vis.
Zürich, Switzerland: Springer, 2014, pp. 184–199.
[45] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using
deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 38, no. 2, pp. 295–307, Feb. 2016.
[46] J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional
network for image super-resolution,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jun. 2016, pp. 1637–1645.
[47] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection
networks for super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit., Jun. 2018, pp. 1664–1673.
[48] I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. Neural
Inf. Process. Syst., 2014, pp. 2672–2680.
Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 60, 2022
[49] J. Caballero et al., “Real-time video super-resolution with spatio-
temporal networks and motion compensation,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 4778–4787.
[50] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in Proc. Int. Conf. Med. Image
Comput. Comput.-Assist. Intervent. Munich, Germany: Springer, 2015,
pp. 234–241.
[51] J. Dai et al., “Deformable convolutional networks,” in Proc. IEEE Int.
Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 764–773.
[52] X. Wang, K. C. K. Chan, K. Yu, C. Dong, and C. C. Loy, “EDVR: Video
restoration with enhanced deformable convolutional networks,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW),
Jun. 2019.
[53] T.-W. Hui, X. Tang, and C. C. Loy, “LiteFlowNet: A lightweight convo-
lutional neural network for optical flow estimation,” in Proc. IEEE/CVF
Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 8981–8989.
[54] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” 2014, arXiv:1409.1556. [Online].
Available: https://arxiv.org/abs/1409.1556
[55] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time
style transfer and super-resolution,” in Proc. Eur. Conf. Comput. Vis.
Amsterdam, The Netherlands: Springer, 2016, pp. 694–711.
[56] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions
for land-use classification,” in Proc. 18th SIGSPATIAL Int. Conf. Adv.
Geographic Inf. Syst. (GIS), 2010, pp. 270–279.
[57] G.-S. Xia et al., “Structural high-resolution satellite image indexing,” in
Proc. ISPRS TC 7th Symp.–100 Years ISPRS, Vienna, Austria, Jul. 2010,
pp. 298–303.
[58] M. Irani and S. Peleg, “Super resolution from image sequences,” in Proc.
10th Int. Conf. Pattern Recognit., vol. 2, Jun. 1990, pp. 115–120.
[59] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor,
“The 2018 pirm challenge on perceptual image super-resolution,” in
Proc. Eur. Conf. Comput. Vis. Workshops (ECCVW), Sep. 2018.
[60] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The
unreasonable effectiveness of deep features as a perceptual metric,”
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018,
pp. 586–595.
[61] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘completely
blind’ image quality analyzer,” IEEE Signal Process. Lett., vol. 20, no. 3,
pp. 209–212, Mar. 2013.
[62] L. Liu, B. Liu, H. Huang, and A. C. Bovik, “No-reference image quality
assessment based on spatial and spectral entropies,” Signal Process.,
Image Commun., vol. 29, no. 8, pp. 856–863, Sep. 2014.
[63] C. Ma, C.-Y. Yang, X. Yang, and M.-H. Yang, “Learning a no-reference
quality metric for single-image super-resolution,” Comput. Vis. Image
Understand., vol. 158, pp. 1–16, May 2017.
[64] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6,
pp. 84–90, May 2017, doi: 10.1145/3065386.
[65] P. Wang, H. Zhang, F. Zhou, and Z. Jiang, “Unsupervised remote sensing
image super-resolution using cycle CNN,” in Proc. IEEE Int. Geosci.
Remote Sens. Symp. (IGARSS), Jul. 2019, pp. 3117–3120.
[66] A. Dosovitskiy et al., “FlowNet: Learning optical flow with convo-
lutional networks,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV),
Dec. 2015, pp. 2758–2766.
Runmin Dong received the bachelor’s degree
in information and computing science from the
Department of Science, Beijing Jiaotong University,
Beijing, China, in 2017. She is currently pursuing
the Ph.D. degree in ecology with the Department of
Earth System Science, Tsinghua University, Beijing.
Her research interests include remote sensing
image processing, deep learning, land cover map-
ping, image super-resolution reconstruction, and
self-supervised representation learning.
 DONG et al.: RRSGAN: RefSR FOR REMOTE SENSING IMAGE 5601117

Lixian Zhang received the bachelor’s degree in Haohuan Fu (Member, IEEE) received the Ph.D.
engineering of surveying and mapping from the degree in computing from Imperial College London,
School of Geodesy and Geomatics, Wuhan Uni- London, U.K., in 2009.
versity, Hubei, China, in 2018. He is pursuing the He is a Professor with the Ministry of Education
Ph.D. degree in ecology with the Department of Key Laboratory for Earth System Modeling and
Earth System Science, Tsinghua University, Beijing, the Department of Earth System Science, Tsinghua
China. University, Beijing, China. He is also the Deputy
His research interests include building extrac- Director of the National Supercomputing Center,
tion from remote sensing images, deep learn- Wuxi, China. His research interests include design
ing, and remote sensing image super-resolution methodologies for highly efficient and highly scal-
reconstruction. able simulation applications that can take advantage
of emerging multicore, many-core, and reconfigurable architectures, and make
full utilization of current Peta-Flops and future Exa-Flops supercomputers; and
intelligent data management, analysis, and data mining platforms that combine
the statistics methods and machine learning technologies.

Authorized licensed use limited to: National Technical University of Athens (NTUA). Downloaded on December 26,2022 at 20:03:24 UTC from IEEE Xplore. Restrictions apply.