Neurocomputing 505 (2022) 178–187

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

Adherent mist and raindrop removal from a single image using attentive
convolutional network
Da He a, Xiaoyu Shang a, Jiajia Luo b,c,⇑
a
University of Michigan-Shanghai Jiao Tong University Joint Institute, Shanghai Jiao Tong University, Shanghai 200240, China
b
Institute of Medical Technology, Peking University Health Science Center, Beijing 100191, China
c
Biomedical Engineering Department, Peking University, Beijing 100191, China

a r t i c l e i n f o a b s t r a c t

Article history: Temperature-difference-induced mist and environmentally-induced raindrops adhering to glass products
Received 25 November 2021 such as windshield and camera lens, can often block vision and severely degrade the image seen through
Revised 27 June 2022 the lens. Despite posing a challenge to various vision systems, including autonomous driving and security
Accepted 12 July 2022
surveillance, this problem has not received sufficient attention from researchers. In this work, we discuss
Available online 16 July 2022
the image degradation caused by adherent mist and raindrops. An attentive convolutional network is
designed to visually remove the adherent mist and raindrops from a single image. Considering the vari-
Keywords:
ations in coexistence and reginal characteristics of adherent mist and raindrops, we propose
Adherent mist removal
Raindrop removal
interpolation-based pyramid attention blocks to perceive spatial information at different scales without
Image restoration rigid priors. Experiments show that the proposed method can improve the visibility of severely degraded
Attention mechanism images in real-world scenarios, both qualitatively and quantitatively. Further application experiments
Convolutional neural network demonstrate that this practical problem is critical to high-level vision situations.
Ó 2022 Elsevier B.V. All rights reserved.

1. Introduction Temperature difference is one of the main causes of adherent

High-level computer vision tasks, such as object detection and
segmentation, are highly dependent on the quality of captured
images. However, outdoor vision systems such as security moni-
toring and vision-based self-driving cars or driving assistance sys-
tems are easily influenced by severe weather or other conditions.
As shown in Fig. 1, raindrops adhering to a camera lens can signif-
icantly degrade the image, and mist in the car adhering to a wind-
shield can reduce visibility.
Adherent raindrops (or waterdrops) exist in a wide range of sce-
narios. Outdoors, rain streaks will result in raindrops. On a fishing
boat, splashing water can form waterdrops. Because the visual
effects from raindrops vary due to factors such as the different sizes
of the drops, their distance to the camera, and complex refraction, it
is extremely difficult to model the raindrops manually.
Another severe and often-observed source of interference with
visibility, adherent mist, is often overlooked in computer vision
research. As shown in Fig. 1(b), mist adhering to a windshield
can severely obstruct a driver’s vision. The scene in Fig. 1(b) is
often encountered when driving a car in cold weather.
⇑ Corresponding author at: Biomedical Engineering Department, Peking Univer-
sity, Beijing 100191, China.
E-mail addresses: da.he@sjtu.edu.cn (D. He), shangxiaoyu@sjtu.edu.cn
(X. Shang), jiajia.luo@pku.edu.cn (J. Luo).
mist. Adherent mist often forms in conjunction with raindrops,
especially when there is a marked temperature difference between
the interior of the car and the windshield or windows cooled by the
raindrops outside. Moreover, dust on the glass surface and the
impurities in water may contribute to the formation of mist-like
stains after the waterdrops evaporate. In addition to the fact that
the interference with visibility caused by the coexistence of adher-
ent mist and raindrops will often be experienced by cars or buses
on rainly days or in cold areas, the degradation is also obvious in
monitoring datasets (such as those from The Nature Conservancy).
Therefore, it is necessary to handle both adherent mist and rain-
drops simultaneously. Fig. 1(c) shows the overlap of mist and rain-
drops adhering to a camera lens.
By improving the formulas in [1–4], the degraded image Icap
captured by the camera can be mathematically described as a com-
bination of raindrops R, the scattering of adherent mist A, and the
clean background B as follows:
Icap ¼ ðð1  MÞ  B þ RÞ  t þ A  ð1  tÞ; ð1Þ
where M denotes a binary mask showing the existence of raindrops,
 indicates element-wise multiplication, and t is the transmission
map indicating information passing ratios through the adherent
mist. Icap ; M; B; R; t, and A are location-related maps (i.e., matrices)
instead of constants.

https://doi.org/10.1016/j.neucom.2022.07.032
0925-2312/Ó 2022 Elsevier B.V. All rights reserved.
D. He, X. Shang and J. Luo
Fig. 1. Examples of adherent raindrops and/or mist. (a) Raindrops adhering to a glass before the camera [1]; (b) Inhomogeneous mist adhering to a windshield when people
sitting in the car; (c) Mist and raindrops simultaneously adhering to a camera lens.
Although disturbances from adherent mist and raindrops are
widely observed in many scenarios, and visibility can be signifi-
cantly deteriorated for algorithms as well as individuals, limited
researches on these natural phenomena have been conducted in
the field of computer vision. Particularly, there seems to be no
studies on adherent mist.
Admittedly, both adherent mist and raindrops on windows can
be cleaned off by defrosters and windshield wipers. However,
algorithm-based solutions and attention to the proposed degrada-
tion are still in demand in many real-world scenarios. For example,
self-driving system should be able to deal with the degradation in
visibility whether or not it can switch on the defrosters and wipers
in a timely fashion. The car’s drive recorder app is expected to work
when other hardware stops. In addition, low-cost vision devices
like monitoring cameras do not have defrosters or wipers.
In this study, we aim to remove the adherent mist and rain-
drops from a single impaired image to obtain a clean image. Based
on convolutional neural networks (CNNs), the proposed approach
can automatically visually remove raindrops and mist to varying
degrees. It can also improve the effectiveness of high-level vision
algorithms in severe weather.
Because raindrops and mist can partially or even completely
block some objects, it is challenging to reproduce a purely clean
image. To solve the problem, we apply the basic blocks in [5] and
advanced attention mechanisms to construct our network. In addi-
tion to the typical convolutional layers, local residual learning,
channel-wise attention, and spatial attention are applied to extract
features. Long-distance shortcuts are also used, forming different
feature stages from corresponding layer groups before feature
fusion. To strengthen the spatial attention facing the variations of
adherent mist and raindrops, an interpolation-based pyramid
attention (IPA) block is applied to every group. The IPA block first
zooms the features to various sizes then perceives spatial attention
from these scaled spaces.
For our experiment, we used a dataset containing degraded and
clean image pairs. Unlike in purely synthetic datasets, the adherent
mist and raindrops in our dataset physically exist and could there-
fore model real-world scenarios. The proposed method can
improve the visibility of degraded images and outperforms other
advanced networks. Moreover, it still exhibits the state-of-the-art
performance when applied to the tasks of removing traditional
atmospheric haze or purely raindrops.
In summary, our main contributions include the following:
1) We address the rarely studied visual interference problem
caused by adherent mist and raindrops. Without appropriate
solutions, this problem limits the extensive use of systems
based on outdoor cameras.
2) We use an experimental-based dataset that contains 1560
image pairs, focusing on the coexistence of adherent mist and
raindrops.
3) An end-to-end restoration network is applied to remove
adherent mist and raindrops without hand-crafted priors. The
179
Neurocomputing 505 (2022) 178–187
proposed IPA contributes to handling images degraded by
regional visibility interferences such as haze, adherent mist,
and raindrops. As a result, our network shows high restoration
performance for the proposed task, as well as the tasks of
removing conventional haze and only raindrops.
2. Related work
2.1. Attention mechanism
Instead of treating all features equally, which is inefficient or
even harmful to the convergence of deep neural networks, visual
attention mechanism usually assigns different attention weights
to different features. Important features for specific tasks desire
strong attention. Common attention mechanisms for CNNs can
be divided into channel-wise attention and spatial attention. The
former has achieved great success in image classification [6] and
was then introduced to image restoration tasks [7]. Based on the
experiments for de-noising, de-mosaicing, artifacts removing and
image super-resolution, Zhang et al. demonstrated the importance
of both attention mechanisms for a wide range of low-level vision
tasks [8].
2.2. Rain streak removal
Most visibility enhancement algorithms related to rain weather
focused on rain streaks [9]. However, the shapes and physical
effects of raindrops differ greatly from those of rain streaks. There-
fore, these streak removal articles may not be directly comparable
to problems involving the removal of raindrops [10].
2.3. Raindrop removal
Several studies on raindrop detection have been undertaken.
Kurihata et al. [11] utilize principal component analysis to learn
the characteristics of raindrops. Ito et al. [12] use the maximally
stable extremal regions to detect raindrop candidates. You et al.
[13] pay attention to both raindrop detection and removal in
videos. They notice that the temporal change of intensity of rain-
drop pixels is different from that of non-raindrop pixels, which
does not apply to the single image case. Eigen et al. [14] may have
been the first to address the problem of single image raindrop
removal by training a shallow CNN with paired data. Unfortu-
nately, the limited capacity of the CNN model restricts its perfor-
mance, especially for large and dense raindrops.
Qian et al.’s work (DeRaindrop) [1] might have been the first
practical solution for the problme of raindrop removal. They build
a generative adversarial network (GAN) [15] to generate raindrop-
free images. Besides the GAN, DeRaindrop includes a recurrent net-
work to gradually locate raindrops in the input image as a spatial
attention. The performance is related to this raindrop mask, and
its ground truth is calculated by the difference between the
degraded image and the clean image with a hand-crafted threshold
D. He, X. Shang and J. Luo
of 30. However, it is difficult to apply this manual threshold to
images with adherent mist, which contain complex pixel-value
variations. Quan et al. [16] propose shape-driven attention to the
raindrop region, which avoids the rigid threshold but introduces
another strong prior about raindrops’ shapes. This prior is also
inapplicable to raindrops with adherent mist, as the edges in the
image may be more severely damaged. By contrast, methods with-
out hard priors might be more useful for comprehensive real-world
situations [17].
2.4. De-hazing
Few studies have primarily focused on adherent mist. Some
similar studies mainly concentrate on atmospheric haze (or fog)
[2,3,18–20]. They both have the scattering effect, which results in
a ‘‘white” color and degrades the original information. However,
compared to atmospheric haze, adherent mist is more likely to
be inhomogeneous as it is related to the specific design of the
equipment, such as which direction the warm steam comes from
and how the window looks (e.g., mist adhering to the windshield
near the driver’s seat in Fig. 1(b)).
Single image de-hazing can be explored using some priors-based
methods. He et al. [20] propose a dark channel prior to discuss the
local minimum of the dark channel that varies between haze and
haze-free images. Yuan et al. [21] unify several existing priors and
design a confidence prior to better handle outliers and noises.
Moreover, some learning-based methods have been proposed.
Cai et al. [18] introduce an end-to-end CNN to estimate the trans-
missions from images with haze. Ren et al. [3] propose a multi-
scale gated fusion network to improve performance. The smoothed
dilation technique is adopted by Chen et al. [2] to remove gridding
artifacts. Qin et al. [5] focus on the utilization of feature fusion and
achieve better de-hazing performance than previous and even
some newer methods with the notable margin.
However, in our work, the superposition of adherent mist and
raindrops results in complex local features, limiting general de-
hazing algorithms using transmission estimation. It is also difficult
to manually input an attention map similar to those in [1,16].
Therefore, the proposed network without explicit priors is
designed to effectively utilize features and restore images for the
real-world scenarios.
3. Method
3.1. Network architecture
The pipeline of our method is shown in Fig. 2. We organize the
basic blocks and feature attention blocks from [5] to build our
Fig. 2. Pipeline of the proposed method. (a) Diagram of the dataset acquisition; (b) Overview of the applied convolutional network; (c) Some legends; (d) The transformation
of feature maps in the IPA block.
180
Neurocomputing 505 (2022) 178–187
network backbone, which is denoted as the baseline. The baseline
contains 114 basic blocks, uniformly distributed in six groups.
There is a long-distance shortcut in each group, which forms the
residual learning mechanism for every 19 basic blocks. The output
feature maps of each group are then fused. A global residual short-
cut is applied so that the entire network learns the transformation
residuals.
Due to the characteristics of adherent raindrops, Qian et al. [1]
and Quan et al. [16] designs prior-based attention mechanisms to
enhance the attention performance for raindrop regions. Combined
with adherent mist, the additional attention becomes more impor-
tant, while those priors hardly hold. Therefore, to further improve
the perception for adherent mist and raindrops, we propose and
insert an IPA block into each group in the baseline model to imple-
ment the final network (denoted as baseline + IPA), as shown in
Fig. 2(b).
3.2. Interpolation-based pyramid attention
Adherent mist and raindrops display patterns that are very dif-
ferent from common human-made objects (such as buildings and
chairs) and widely-studied noises (such as Gaussian noise and rain
streaks). Artificial objects usually have rigid shapes. The convolu-
tional kernels are sensitive to their low-level edges and high-
level shapes. Conventional noises are always discretely distributed
in the image, and most background information remains. However,
adherent mist has non-rigid edges and attenuates the sharpness of
adherent raindrops in images. The adherent raindrops, in turn,
aggravate the visual inhomogeneity. Complex refraction and scat-
tering occurs when adherent mist and raindrops coexist, causing
much more severe visual degradation than general atmospheric
haze and purely raindrops.
Therefore, additional techniques to help the CNN extract fea-
tures from the non-rigid and severe degradation would be useful.
Unlike other pyramid attention mechanisms that fuse different lay-
ers [22] or adopt different convolutional kernels for the same layer
[23–25], our IPA generates different scaling features by bilinear
interpolations, as shown in Fig. 2(d). Specifically, for feature maps
from the same layer, IPA first amplifies or shrinks them by interpo-
lation. Two convolutional layers are successively applied at each
scale to obtain three attention maps with different scales. Finally,
attention maps are interpolated back to the original lateral size
and added together. The final spatial attention map is then adopted
to weight the original feature map by pixel-wise multiplication.
Like other pyramid attention techniques, IPA learns spatial
attention based on features with different scales, enabling large
receptive fields and flexible perception for patterns of various
sizes. Moreover, the interpolation operation outperforms
D. He, X. Shang and J. Luo Neurocomputing 505 (2022) 178–187

convolution-based upsampling/downsampling (e.g., [26,22]) for 4. Experiment

another advantage in this task. Because the bilinear interpolation
can be regarded as the low-pass filter when processing images,
high-frequency information in the feature map is suppressed after
interpolation. In the original feature map, the background textures
in clean regions usually correspond to high-frequency information.
In contrast, textures obscured by adherent mist and raindrops
mainly contain relatively low-frequency signals. Therefore, after
interpolation (i.e., the low-pass filter), the attention mechanism
is less likely to pay meaningless attention to clean regions, and
areas covered by adherent mist and raindrops could possibly
obtain more attention than before. The conclusion is that the IPA
block may help to strengthen the attention mechanism for regional
visibility interference problems.
3.3. Loss function
As indicated in [27,28], although pixel-wise mean squared error
(MSE) or mean absolute error (MAE) may bring significant
improvements to metrics such as peak signal-to-noise (PSNR)
and structural similarity index (SSIM) [29], the generated image
is likely to be over-smoothed. Instead, a perceptual loss term can
be applied to calculate the error in high-level views, which might
help generate subjectively realistic images.
We calculate the perceptual loss based on the MSE of the output
features from the third block of VGG16 [28]. The VGG16 model is
pretrained on the ImageNet dataset [30], which may provide a rel-
atively high-level description of an image. Finally, the perceptual
loss is added together with the general MAE loss:
Ltotal ¼ k1 Lmae þ k2 Lper ; ð2Þ
where k1 ; k2 are the weight coefficients of items, Ltotal is the total
loss, Lper denotes the perceptual loss, and Lmae is calculated by
directly comparing the prediction and the ground truth using MAE.
3.4. Data collection
To collect the image pairs consisting of clean and degraded
images, we take photographs following strategies inspired by [1].
The schematic is simplified in Fig. 2(a). Specifically, we utilize
two of the same glass panels to simulate different cases. First, we
use a tripod to secure the camera. Second, one of the panels is ran-
domly sprinkled with waterdrops on one side and randomly
sprayed with mist from a cosmetic sprayer on the other side. Then,
we place the blurred panel in front of the camera and take a photo
of the degraded image. Finally, the corresponding ground truth
(i.e., clean) image is taken with the other clean glass panel.
To maintain good generalization, the distributions of water-
4.1. Experiment setup
The proposed method is implemented using the PyTorch frame-
work and trained with an Nvidia Titan RTX GPU. We use Adam
optimizer with a learning rate of 1  104 . The patch-based train-
ing strategy is applied with the cropped patch size of 240  240
and the batch size of 2. Other data augmentation operations
including flipping and rotation are also used. The coefficients
k1 ; k2 in the loss function are 1:00 and 0:04, respectively.
Patch-based training is often used to improve training perfor-
mance, while too small patch size might damage the extraction
of degradation features. The loss coefficents are used to weight
and balance different loss terms. Too large perceptual loss might
impair the convergence in the early stages of training. As a result,
the selection of the above hyper-parameters is generally
experiment-based. We have conducted a parameter searching to
obtain the listed learning rate, patch size and loss coefficients.
4.2. Restoration results
To our knowledge, there have been no other comparable studies
on the problem of jointly handling adherent mist and raindrops.
Therefore, we compare the proposed method with the following
three compromised aspects.
1) Our method is compared with two two-step combinations. A
two-step combination indicates we successively adopted a pre-
trained de-hazing approach (that originally focuses on atmo-
spheric haze) and a pre-trained raindrop removal model to
restore the image. In other words, the degraded image is first
processed by a de-hazing model (i.e., GCANet by Chen et al.
[2], or FFANet by Qin et al. [5]) and then by a raindrop removal
model (i.e., Qian et al. [1] or Quan et al. [16]).
2) We also tried to apply raindrop removal models before GCA-
Net (or FFANet) for comparison.
3) Although originally meant for other tasks, the methods from
DeRaindrop [1], GCANet [2], FFANet [5] and Quan et al. [16] are
advanced restoration networks that can technically be re-
trained on our dataset, achieving the joint removal of adherent
mist and raindrops to some extent.
The results shown in Table 1 indicate that, first of all, it is
important to handle adherent mist and raindrops together. The
coexistence of mist and raindrops may lead to results that are even
Table 1
Quantitative comparisons of different methods on our dataset for adherent mist and
raindrops removal tasks.

drops and mist are random. Moreover, the distance between the Method PSNR SSIM
camera and the glass panel is also a random value from 0.5 cm Degraded Image 17.84 0.6048
to 4.0 cm. Thus the shape of waterdrops could appear differently
Chen et al. [2] + Qian et al. [1] 15.12 0.5919
in the image due to camera focal length. A Canon EOS 800D camera Qin et al. [5] + Qian et al. [1] 17.57 0.6302
and the integrated camera of the Redmi K20 Pro samrtphone are Chen et al. [2] + Quan et al. [16] 14.75 0.5670
used for taking pictures. Qin et al. [5] + Quan et al. [16] 17.30 0.6100
We alleviate two dataset issues in [1] during our preparation: Qian et al. [1] + Chen et al. [2] 16.30 0.6167
the observation of unexpected reflection images on the glass Qian et al. [1] + Qin et al. [5] 18.83 0.6383
plates, and the fact that many samples in [1] were acquired on Quan et al. [16] + Chen et al. [2] 15.96 0.5817
Quan et al. [16] + Qin et al. [5] 18.48 0.6114
sunny days.
Finally, all the collected images are resized with bicubic inter- Re-trained Qian et al. [1] 18.89 0.7012
Re-trained Chen et al. [2] 22.88 0.7968
polation to the shape of 640  480 to form the dataset. There are
Re-trained Qin et al. [5] 23.45 0.8056
1248 image pairs for training, 156 pairs for validation, and 156 Re-trained Quan et al. [16] 24.12 0.8089
pairs for testing. All the following evaluation results are based on
Ours 24.66 0.8293
images in the testing set.
181
D. He, X. Shang and J. Luo
worse than the degraded images (e.g., Chen et al. + Qian et al.
results in Table 1). The four re-trained models generally show
effectiveness, especially for re-trained Chen et al., re-trained Qin
et al. and re-trained Quan et al., but our method shows the best
performance.
In realistic scenes, the combination of adherent mist and rain-
drops might severely impair the image, blocking many objects
from the subjective view. Examples in Fig. 3 exhibit the poor shoot-
ing environments and the restoration results from several of the
above methods. Due to space limitations, we present only five rel-
atively high-performance methods in Fig. 3 as the compared
methods.
As shown in Fig. 3, some objects are not distinguishable in the
degraded image. For example, in the first row of Fig. 3, the decora-
tive text on the pink wall is invisible in the degraded image. By
contrast, most words can be distinguished in the restoration result
from our method. A similar phenomenon is shown in the second
row. The key piece of information in the captured image, the house
number, is recovered by our method. For the third sample, our
result’s general color is intuitively closest to that in the ground
truth. The example in the fourth row can be regarded as a traffic
control application. The camera in the parking lot is severely influ-
enced by the poor weather conditions and takes degraded photos
as shown in the picture. In this case, high-level algorithms may
not recognize the plate number and guards cannot easily confirm
the bus position. However, our method can recover the plate num-
bers on the buses and determine the condition of the parking lot at
a glance.
The comparison reveals that the proposed method can obtain
relatively cleaner images and fewer artifacts than the other meth-
ods; this is also proved through PSNR/SSIM values. In addition, the
combination method (i.e., Qian et al. [1] + Qin et al. [5]) in Fig. 3
shows a tiny improvement. Our method also displayed robust per-
formance for various conditions, and is able to handle relatively
weak interference with visibility (e.g., the third sample in Fig. 3)
Fig. 3. Intuitive comparisons among a few restoration methods. From left to right: ground truth, degraded image, Qian et al. + Qin et al. [1,5], re-trained Chen et al. [2], re-
trained Qin et al. [5], re-trained Quan et al. [16] and our method. Each row corresponds to the same sample. The values below images (except ground truth images) indicate
the corresponding values of PSNR / SSIM. The details of the images are best viewed by zooming in.
182
Neurocomputing 505 (2022) 178–187
and relatively severe degradation (e.g., the other three samples in
Fig. 3). The fact that the color of the cement floor in the fourth sam-
ple in Fig. 3 is a similar color to the mist does not affect our meth-
od’s performance.
Therefore, the severe degradation of visibility from adherent
mist and raindrops is observed. Unlike the well-studied restoration
problems of de-noising or motion blur removal, the new task above
is much more complicated, and it is hard to estimate the degrada-
tion kernel. Simply re-training restoration networks leaves many
artifacts. Our method outperforms the compared advanced net-
works, quantitatively as well as intuitively.
4.3. Extended experiments
Besides the newly proposed vision task described above, we
also extend our network to other two conventional problems to
show its broad applicability. Specifically, the proposed network is
re-trained for atmospheric de-hazing and purely adherent raindrop
removal.
4.3.1. De-hazing
Without any modification to the architecture, we also train our
network using the RESIDE dataset, a widely used de-hazing bench-
mark. Following the same operations as previous studies, we use
the indoor set containing over 13 K image pairs for training and
the indoor synthetic objective testing set (SOTS), including 500
pairs for testing. The testing results are listed in Table 2. For fair-
ness and convenience, the performance values of other methods
are cited from [5]. In addition, three samples from the testing set
are shown in Fig. 4 for qualitative comparison. Only our method
and the two relatively new advanced approaches, Chen et al. [2]
and Qin et al. [5], are presented.
As shown in Fig. 4, both the previous works and our method
achieve nearly perfect performance. However, as shown in Table 2,
the proposed method outperforms other methods in the PSNR and
D. He, X. Shang and J. Luo Neurocomputing 505 (2022) 178–187

Table 2 generating attention masks using the shape-driven method [16]

Quantitative comparisons of methods for de-hazing. is also compared. Similarly, the metric values of compared meth-
Method PSNR SSIM ods are directly cited from [1,16]. Three examples from the testing
He et al. [20] 16.62 0.8179 set are used for qualitative comparison in Fig. 5.
Li et al. [31] 19.06 0.8504 As shown in Table 3, our method achieves comparable results
Cai et al. [18] 21.14 0.8472 with Qian et al. [1] and Quan et al. [16] using the same test set.
Ren et al. [3] 22.30 0.8800 Although our PSNR value is slightly lower than that of [1] or
Chen et al. [2] 30.23 0.9800
Qin et al. [5] 36.39 0.9886
[16], our method has the highest SSIM value.
A better qualitative performance of our method can be seen in
Ours 40.02 0.9931
Fig. 5. By comparing the walls of the first sample in Fig. 5, it is easy
to see that some light ‘‘shades” remain in Qian et al.’s result, which
correspond to the original raindrops’ locations. In contrast, our
method does a better job in terms of color consistency. However,
based on the remaining two samples in Fig. 5, artifacts are still evi-
dent in the zoomed subplots when using methods from [1,16],
while our method also achieves reasonable results in these regions.
The two above issues demonstrate two advantages of our
method, which avoids explicit priors in [1,16]. The work of Qian
et al. [1] sets a threshold of 30 to annotate the raindrop mask for
training, which is a hand-crafted prior. This prior is not suitable
for scenes with monochromatic backgrounds on cloudy or rainy
days. In these scenes, the pixel difference between raindrop
regions and clean regions may become so small that the prior does
not work, and the performance of [1] decreases (see the three sam-
ples in Fig. 5). [16] applies a complex shape-driven prior to gener-
ate the raindrop mask. However, even without adherent mist,
raindrops are still very difficult to model. For image regions with
dramatic gradient changes (e.g., the zoom-in areas of the second
and third samples in Fig. 5), [16] may result in severe artifacts.
Therefore, with similar quantitative performance, our method
without hand-crafted priors is hopefully more effective for practi-
cal scenes.

4.3.3. Summary of the extended experiments

Fig. 4. Qualitative comparisons for three de-hazing methods. From top to bottom:
ground truth, hazy image, Chen et al. [2], Qin et al. [5], and our method. Each
column corresponds to the same sample.
Although the primary purpose of this study is to call for atten-
tion on the newly proposed problem of adherent mist and raindrop
removal, the proposed method works well not only on our own
dataset containing adherent mist, but also for conventional de-
hazing and purely raindrop removal problems. In other words,
these tasks can be handled in applications using only one method.

SSIM metrics. Therefore, the proposed method could be applied to 4.4. Ablation study
handle the conventional de-hazing problem with appropriate
training. Our network’s architecture can be divided into two parts: the
baseline and the IPA blocks. Therefore, we investigate the influence
4.3.2. Sole raindrop removal of the existence of IPA blocks concerning the above problems. In
DeRaindrop (Qian et al. [1]) is a state-of-the-art approach for addition, we compare our IPA with a convolution-based pyramid
handling adherent raindrops. A dataset with purely raindrop attention (CPA) block and a pyramid pooling (PP) feature fusion
degraded image and clean image pairs was also published by Qian block [33]. CPA is implemented by simply removing interpolations
et al. [1]. Therefore, we also try our method using that dataset to from IPA and modifying convolution parameters for rescaling.
verify our approach’s feasibility when only handling adherent rain- Moreover, baseline + PP is achieved by replacing the entire IPA
drops. The dataset contains 861 image pairs for training and 58 block with the pyramid pooling in Zhang et al. [33] and slightly
image pairs for quantitative evaluation. We still do not modify modifying the number of channels. The results of the ablation
the architecture of our method. The quantitative performance is study are shown in Table 4.
shown in Table 3. In addition to [1], a state-of-the-art method for As shown in Table 4, the proposed IPA block outperforms the
two existing pyramid architectures in the three restoration tasks,
which is reasonable because the IPA block can combine multi-
Table 3
Quantitative comparisons of methods on the test set in [1] for solely adherent
scale learning and the low-frequency attention mechanism. Table 4
raindrop removal. also provides quantitative evidence of the contribution of the IPA
block to the performance improvement based on the baseline
Method PSNR SSIM
model in the three experiments. Moreover, although higher perfor-
Eigen et al. [14] 28.59 0.6726 mance can be obtained, IPA does not add much computational
Isola et al. [32] 30.14 0.8299
Qian et al. [1] 31.57 0.9023
complexity as six IPA blocks only add 0.01 M parameters (0.11%)
Quan et al. [16] 31.44 0.9263 and 0.97G FLOPs (0.19%) to the model. By contrast, PP adds
Ours 31.33 0.9297
0.24 M parameters (2.71%) and 13.56G FLOPs (2.69%) based on
the baseline but with less improvement.
183
D. He, X. Shang and J. Luo Neurocomputing 505 (2022) 178–187

Fig. 5. Qualitative comparisons for three raindrop removal methods. From top to bottom: ground truth, image with raindrops, Qian et al. [1], Quan et al. [16], and our method.
Each column corresponds to the same sample. Artifacts are best compared by zooming in.

Table 4
Quantitative results and complexity analysis of the ablation study for three tasks.

Methods Proposed task De-hazing Raindrop removal # params FLOPs

PSNR SSIM PSNR SSIM PSNR SSIM
Baseline 24.13 0.8142 36.54 0.9895 30.82 0.9247 8.87 M 504.04G
Baseline + CPA 24.23 0.8226 34.62 0.9849 31.04 0.9238 8.93 M 504.66G
Baseline + PP 24.28 0.8250 35.58 0.9866 31.07 0.9216 9.11 M 517.60G
Baseline + IPA 24.66 0.8293 40.02 0.9931 31.33 0.9297 8.88 M 505.01G

4.5. Visualization of the attention map
To better demonstrate the difference between our IPA with
other attention mechanisms, we visualize the attention maps of
three attention mechanisms as follows. Specifically, in addition
to our IPA, we obtained the ideal attention map (ground truth of
the predicted attention map) in [1] with a threshold of 30, which
is specially designed for raindrop removal tasks. We also analyze
the convolution-based pyramid attention, denoted as CPA before,
which may be of general interest for all restoration tasks. Two visu-
alized examples can be found in Fig. 6.
As shown in Fig. 6, the ideal attention maps of Qian et al. [1] are
clean and can capture some large raindrop regions. However, such
a hard hand-crafted prior attention shows low performance for
regions with monochromic backgrounds. According to the first
example, we can easily find that raindrops in the ‘‘road” and ‘‘lawn”
regions are not distinguished by the attention map of [1]. In con-
trast, the other two adaptive attention maps do not show this defi-
ciency. This phenomenon validates our discussion in the extended
experiments.
Although the attention maps from CPA and our IPA both focus
on severely degraded regions with various backgrounds, our
method shows better and finer activation. In the second example,
184
both CPA and our IPA indicate the left half of the image is more
severely degraded than the right half. However, CPA can not distin-
guish between degraded regions with or without raindrops. In
other words, the small region with mist only and the small region
with overlapping mist and raindrops obtain similar attention
intensity by CPA. In contrast, the attention map by IPA can well dis-
tinguish three different kinds of regions: low intensity for clean
regions, medium intensity for regions degraded by mist only, and
high intensity for regions degraded more severely. Thus, the above
observations demonstrate the superiority of the proposed IPA for
the proposed task.
4.6. Application experiment
In addition to the above quantitative and qualitative evaluation
results, the effectiveness of our method is also validated in practi-
cal applications. Panoptic segmentation is a newly proposed seg-
mentation task [34] that tries to label all the image pixels. It is
an essential high-level task in many areas, including autonomous
cars. The segmentation results based on ground truth images,
degraded images, and restoration images are shown in Fig. 7.
Specifically, we applied the panoptic segmentation algorithm,
R101-FPN [35] from Detectron2 Model Zoo ( https://
D. He, X. Shang and J. Luo Neurocomputing 505 (2022) 178–187

Fig. 6. Visualization of three attention mechanisms. From left to right: clean image (ground truth), degraded image, ideal attention map of Qian et al. [1], adaptively predicted
attention map of baseline + CPA model, and adaptively predicted attention map of our model. The first row corresponds to an example with little adherent mist but obvious
raindrops, while the second row shows another example with obvious mist and raindrops. All attention maps share the same colorbar.

Fig. 7. Restoration results from different methods and their panoptic segmentation results. From left to right: ground truth, degraded images, Qian et al. + Qin et al. [1,5], re-
trained Qian et al. [1], re-trained Chen et al. [2], re-trained Qin et al. [5], and our method. Each two rows correspond to the same sample. The panoptic segmentation result is
attached below the corresponding image. The details of the images are best viewed by zooming in.

github.com/facebookresearch/detectron2), to segment these
images.
As shown in Fig. 7, panoptic segmentation might make it easy to
show the influence of adherent mist and raindrops from the per-
spective of vision algorithms. When observing the degraded image
in the first sample, the segmentation algorithm cannot find all the
bicycles and is not sure what is behind the bicycles. Undoubtedly,
the uncertainty may influence the decisions involved in self-
driving. Fortunately, bicycles could be counted correctly from
images obtained by the last four methods and the fence could be
roughly located in our restoration result. The second sample shows
a semi-indoor environment. The segmentation borders are not
clear in the degraded image. Simultaneously, the last four methods
could restore the image to the extent that the high-level algorithm
can approximately locate the edges of different surfaces.
Therefore, the results of the application experiment indicate
that mist and raindrops adhering to camera lenses or windshields
can severely hamper high-level computer applications. Moreover,
solving this problem by creating new segmentation datasets is dif-
ficult because humans are also unable to accurately annotate
objects in these degraded images. The proposed algorithm-based
solution might help ease the problem and make the visual devices
more robust.
4.7. Discussion
The above experimental results indicate both the effectiveness
of our method and the importance of the proposed problem. Unlike
previous image restoration problems, the challenge of the new task
185
is magnified because it must jointly handle adherent mist and rain-
drops, but our method handles it well. Specifically, the challenges
are as follows.
1) The image degradation is severe. By intuitively comparing
the degraded examples in Fig. 3 with those in Figs. 4 and 5,
we can easily conclude that the coexistence of adherent mist
and raindrops tends to impair more information than purely
atmospheric haze or only raindrops. The different quantitative
results of different tasks also validate this sense. As a result, a
deep network capable of high performance is used in this work.
Admittedly, although effective, the large amount of computa-
tion required may limit real-time applications. A lightweight
architecture with comparable performance is expected in the
future.
2) Prior-based methods (e.g., [1,16]) do not hold up well. Due to
the interaction between adherent raindrops and adherent mist,
the threshold-based and shape-based prior attention in [1,16]
do not work well in practical scenarios with both degradation
sources. Our method specifically targets at this challenge and
enhance the self-adaptive attention mechanisms by introducing
IPA blocks in addition to common attention layers.
5. Conclusion
In this work, we proposed an image degradation problem
related to the coexistence of adherent mist and raindrops. Both
sources of interference with visibility are adherent to a camera lens
or car windshield, showing different image characteristics com-
D. He, X. Shang and J. Luo
pared with atmospheric haze or rain streaks. Although the problem
is widely observed in real life, little works have been conducted on
it. To solve this problem, we adopted an attentive convolutional
network containing interpolation-based pyramid attention blocks
to strengthen the spatial perception of regional obstacles. Both
quantitative evaluation and intuitive examples showed quality
improvement with the proposed method without explicit priors.
The superiority of the proposed architecture was also validated
even for the tasks of conventional de-hazing and the removal of
purely raindrops. Restoration from the adherent mist and raindrop
degradation can benefit high-level tasks, such as panoptic segmen-
tation, by producing reliable results, which demonstrates the sig-
nificance of the proposed architecture in vision applications. In
the future, we will further simplify the architecture to reduce the
computation complexity while keeping similar performance, thus
the real-time restoration could be achieved in more embedded sce-
nes to handle the proposed adherent mist and raindrop degrada-
tions. In addition, our experimentally acquired dataset and our
codes will be released online (link) to facilitate further studies on
related tasks.
6. CRediT authorship contribution statement
Da He: Conceptualization, Methodology, Software, Validation,
Writing - Original Draft. Xiaoyu Shang: Investigation, Data Cura-
tion. Jiajia Luo: Conceptualization, Methodology, Writing - Review
& Editing, Supervision.
CRediT authorship contribution statement
Da He: Conceptualization, Methodology, Software, Validation,
Writing - original draft. Xiaoyu Shang: Investigation, Data cura-
tion. Jiajia Luo: Conceptualization, Methodology, Writing - review
& editing, Supervision.
Data availability
We have shared the link to the data/code in the section of
Conclusion.
Declaration of Competing Interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Acknowledgements
This work was supported by National Key R&D Program of
China (Grant No. 2021YFF0502900), National Natural Science
Foundation of China (Grant No. 31870942); Peking University Clin-
ical Medicine Plus X – Young Scholars Project (Grant No.
PKU2020LCXQ017, PKU2021LCXQ028); the Fundamental Research
Funds for the Central Universities (Grant No. BMU2021YJ009); and
PKU-Baidu Fund (Grant No. 2020BD039).
References
[1] R. Qian, R.T. Tan, W. Yang, J. Su, J. Liu, Attentive generative adversarial network
for raindrop removal from a single image, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2018, pp. 2482–
2491.
[2] D. Chen, M. He, Q. Fan, J. Liao, L. Zhang, D. Hou, L. Yuan, G. Hua, Gated context
aggregation network for image dehazing and deraining, in: IEEE Winter
Conference on Applications of Computer Vision (WACV), 2019, pp. 1375–1383.
186
Neurocomputing 505 (2022) 178–187
[3] W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, M.-H. Yang, Gated fusion network
for single image dehazing, in: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2018, pp. 3253–3261.
[4] S. Li, I.B. Araujo, W. Ren, Z. Wang, E.K. Tokuda, R.H. Junior, R. Cesar-Junior, J.
Zhang, X. Guo, X. Cao, Single image deraining: A comprehensive benchmark
analysis, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2019, pp. 3838–3847.
[5] X. Qin, Z. Wang, Y. Bai, X. Xie, H. Jia, Ffa-net: Feature fusion attention network
for single image dehazing, in: Proceedings of the AAAI Conference on Artificial
Intelligence, Vol. 34, 2020, pp. 11908–11915.
[6] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the
IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–
7141.
[7] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu, Image super-resolution using very
deep residual channel attention networks, in: Proceedings of the European
conference on computer vision (ECCV), 2018, pp. 286–301.
[8] Y. Zhang, K. Li, B. Zhong, Y. Fu, Residual non-local attention networks for image
restoration, in: International Conference on Learning Representations, 2019.
[9] F. Yang, J. Ren, Z. Lu, J. Zhang, Q. Zhang, Rain-component-aware capsule-gan for
single image de-raining, Pattern Recogn. 108377 (2021).
[10] X. Jin, Z. Chen, W. Li, Ai-gan: Asynchronous interactive generative adversarial
network for single image rain removal, Pattern Recogn. 100 (2020) 107143.
[11] H. Kurihata, T. Takahashi, I. Ide, Y. Mekada, H. Murase, Y. Tamatsu, T. Miyahara,
Rainy weather recognition from in-vehicle camera images for driver
assistance, IEEE Proceedings. Intelligent Vehicles Symposium 2005 (2005)
205–210.
[12] K. Ito, K. Noro, T. Aoki, An adherent raindrop detection method using mser, in:
Asia-Pacific Signal and Information Processing Association Annual Summit and
Conference (APSIPA), 2015, pp. 105–109.
[13] S. You, R.T. Tan, R. Kawakami, Y. Mukaigawa, K. Ikeuchi, Adherent raindrop
modeling, detectionand removal in video, IEEE Trans. Pattern Anal. Mach.
Intell. 38 (9) (2015) 1721–1733.
[14] D. Eigen, D. Krishnan, R. Fergus, Restoring an image taken through a window
covered with dirt or rain, in: Proceedings of the IEEE International Conference
on Computer Vision, 2013, pp. 633–640.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A.
Courville, Y. Bengio, Generative adversarial nets, Advances in Neural
Information Processing Systems (2014) 2672–2680.
[16] Y. Quan, S. Deng, Y. Chen, H. Ji, Deep learning for seeing through window with
raindrops, in: Proceedings of the IEEE International Conference on Computer
Vision, 2019, pp. 2463–2471.
[17] M. Shao, L. Li, H. Wang, D. Meng, Selective generative adversarial network for
raindrop removal from a single image, Neurocomputing 426 (2021) 265–273.
[18] B. Cai, X. Xu, K. Jia, C. Qing, D. Tao, Dehazenet: An end-to-end system for single
image haze removal, IEEE Trans. Image Process. 25 (11) (2016) 5187–5198.
[19] R. Fattal, Single image dehazing, ACM Trans. Graph. (TOG) 27 (3) (2008) 1–9.
[20] K. He, J. Sun, X. Tang, Single image haze removal using dark channel prior, IEEE
Trans. Pattern Anal. Mach. Intell. 33 (12) (2010) 2341–2353.
[21] F. Yuan, Y. Zhou, X. Xia, X. Qian, J. Huang, A confidence prior for image
dehazing, Pattern Recogn. 119 (2021) 108076.
[22] Z.-L. Ni, G.-B. Bian, G.-A. Wang, X.-H. Zhou, Z.-G. Hou, H.-B. Chen, X.-L. Xie,
Pyramid attention aggregation network for semantic segmentation of surgical
instruments, in: Proceedings of the AAAI Conference on Artificial Intelligence,
Vol. 34, 2020, pp. 11782–11790.
[23] H. Li, P. Xiong, J. An, L. Wang, Pyramid attention network for semantic
segmentation, in: Proceedings of the British Machine Vision Conference, 2018.
[24] Z. Huang, Z. Zhong, L. Sun, Q. Huo, Mask r-cnn with pyramid attention network
for scene text detection, in: 2019 IEEE Winter Conference on Applications of
Computer Vision (WACV) IEEE, 2019, pp. 764–772.
[25] H. Wang, G. Wang, Z. Sheng, S. Zhang, Automated segmentation of skin lesion
based on pyramid attention network, in: International Workshop on Machine
Learning in Medical Imaging, Springer, 2019, pp. 435–443.
[26] Y. Mei, Y. Fan, Y. Zhang, J. Yu, Y. Zhou, D. Liu, Y. Fu, T.S. Huang, H. Shi, Pyramid
attention networks for image restoration, arXiv preprint arXiv:2004.13824
(2020).
[27] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and
super-resolution, European Conference on Computer Vision (2016) 694–711.
[28] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A.
Tejani, J. Totz, Z. Wang, et al., Photo-realistic single image super-resolution
using a generative adversarial network, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2017, pp. 4681–4690.
[29] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment:
from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4)
(2004) 600–612.
[30] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale
hierarchical image database, IEEE Conference on Computer Vision and Pattern
Recognition (2009) 248–255.
[31] B. Li, X. Peng, Z. Wang, J. Xu, D. Feng, Aod-net: All-in-one dehazing network, in:
Proceedings of the IEEE international conference on computer vision, 2017, pp.
4770–4778.
[32] P. Isola, J.-Y. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with
conditional adversarial networks, in: Proceedings of the IEEE conference on
computer vision and pattern recognition, 2017, pp. 1125–1134.
[33] H. Zhang, V.M. Patel, Densely connected pyramid dehazing network, in:
Proceedings of the IEEE conference on computer vision and pattern
recognition, 2018, pp. 3194–3203.
D. He, X. Shang and J. Luo Neurocomputing 505 (2022) 178–187

[34] A. Kirillov, K. He, R. Girshick, C. Rother, P. Dollár, Panoptic segmentation, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2019, pp. 9404–9413.
[35] Z. Cai, N. Vasconcelos, Cascade r-cnn: High quality object detection and
instance segmentation, IEEE Trans. Pattern Anal. Mach. Intell. (2019).
Da He received his B.S. degree in Optoelectronic Infor-
mation Science and Engineering from Nankai Univer-
sity. In 2018, he joined the University of Michigan-
Shanghai Jiao Tong University Joint Institute, Shanghai
Jiao Tong University, Shanghai, China, as a graduate
student. He is interested in image processing and deep
learning.
Jiajia Luo received his Ph.D. degree in mechanical
engineering from the University of Michigan, Ann Arbor,
USA. He is currently an Assistant Professor at the
Biomedical Engineering Department, Peking University,
Beijing, China. His research interests include biomedical
imaging, biomechanics, and machine learning. He was a
recipient of a career development award in Michigan
Institute for Clinical & Health Research, and Peking
University Clinical Medicine + X Young Investigator
Grant Award. His research is supported by National Key
R&D Program of China, National Natural Science Foun-
dation of China, and PKU-Baidu Fund.

Xiaoyu Shang received the B.S. degree in Optoelectronic

Information of Science and Engineering from Sun Yat-
sen University. In 2018, she joined the University of
Michigan-Shanghai Jiao Tong University Joint Institute,
Shanghai Jiao Tong University, Shanghai, China, as a
graduate student. Her research interest is image pro-
cessing, especially for optical imaging with deep learn-
ing.

187