NOISE ROBUST SINGLE IMAGE SUPER-RESOLUTION USING A MULTISCALE IMAGE
PYRAMID
Jing Hu*, Jiaxin Li, Xi Wu, Jiliu Zhou
Department of Computer Science, Chengdu University of Information Technology, P.R. China, 610225
ABSTRACT
Single image super-resolution (SR) generates a high-
resolution (HR) image by estimating the mapping function
between image patches of different resolutions. However,
this kind of SR method cannot be directly applied to noisy
images, since noise will be reinforced in the process of
super-resolution. To this end, this paper presents a
simultaneous super-resolution and denoising method by
exploiting the noise decreasing property contained in the
multiscale image pyramid. Experimental results confirm that
our method is able to outperform other state-of-the-art super-
resolution methods when super-resolving noisy images
across differing noise levels.
Index Terms—single-image super-resolution, noise-
robust, multiscale image, self-similarity
1. INTRODUCTION
Single image super-resolution (SR) aims to generate a high
resolution (HR) image from a single input low resolution
(LR) image. The key problem of SR is to synthesize the
high-frequency (HF) details which are lost in the LR image.
Some single image SR methods [1-4], or called learning-
based SR methods, learn such details from a set of external
LR-HR patch pairs, while others[5-8], named as self-
learning-based SR (SLSR) methods, use the LR input image
itself as the only source of LR-HR example patches. The
common idea of both types of SR approaches is more or less
the same. That is, to infer an HR image or HR patch by
learning the relationship between LR-HR exemplar patch
pairs. However, utilizing the external training images,
learning-based SR methods are very likely to introduce
irrelevant HF details if the LR input image is incompatible
with the training datasets [1, 5]. On the contrary, based on
the observation that local image structures tend to reappear
within and across different image scales [7], SLSR
approaches regularize the HF detail estimation problem on
these self-examples, and therefore circumvent the false-
detail issue.
Although SLSR avoids false-detail problem, its
fundamental drawback is that only the HF details that have
978-1-4799-7061-2/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY TAXILA. Downloaded on May 17,2023 at 06:35:58 UTC from IEEE Xplore. Restrictions apply.
already existed in the original LR image can be enhanced.
Therefore, a fundamental problem of SLSR is how to
enhance HF details properly. Inappropriate HF enhancement
is not only at the risk of producing overshoot artifacts
around edges in SR result, but also amplifies LR image noise
as well. However, although several researchers [8, 9] have
pointed out the noise amplification problem of SLSR
framework, surprisingly, to the best of our knowledge, a
very limited number of SLSR works have seriously
addressed this issue. Typically, when dealing with noisy
images, most SLSR methods involve a two-step process:
super-resolution and denoising. Unfortunately, Singh et al.
[10] have acclaimed that such two-step SLSR framework
would reinforce the spatial correlation of the original image
noise by the initial SR process, and the following denoising
step cannot remove such noise effectively.
To this end, a noise-robust SLSR framework is studied
in this paper. Based on observation that the interpolation
process embedded in conventional SLSR accounts for noise
amplification, we propose a simultaneous interpolation and
denoising method and combine it to the SLSR framework.
Fig. 1(b) is the flowchart of the proposed method.
The remainder of this paper is structured as follows. In
Section 2, we first briefly introduce the SLSR framework
and then illustrate the proposed simultaneous interpolation
and denoising method. Experimental results on several noisy
images are provided in Section 3, together with visual and
quantitative comparisons with other methods. Finally, the
conclusions are described in Section 4.
2. METHOD
We develop our combined SLSR method based on our
previous work on SLSR [11]. We firstly introduce the SLSR
framework and explain the problem of directly applying
such framework to noisy LR image. Then we summarize our
observation and propose our combined SR framework to
address this problem.
For convenience, the variables used in the proposed
method are defined as follows: as shown in Fig. 1(a),
matrices I (I0) and H are used to represent the input LR
image and the output HR image, I0,s is the smoothed version
2526 ICIP 2018
 of I0 and I1,s is an interpolated version of I0. The subscript t q  f qs   f ps  qs  ps 
in I   denotes t-th iteration of the algorithm. The bolded
t
1 (1)
 f  p s   f   p s   q s  p s   f   p s   q s  p s   q s  p s  
lowercase p and q terms denote the column vectors of two 2
s×s image patches which are extracted from I 0  and I1  ,
t t
where denotes the element-wise product, f    and
respectively; ps and qs are the column vectors of two s×s f    denote the first- and second derivatives of the
image patches taken from I 0,t s and I1, t s , respectively. Without regression function. Moreover, by assuming the input LR
specific notification, we assume that p, ps, q, and qs are image is noise-free and the mapping function follows some
related. That is, ps is the most similar among all patches to smoothness property, we come to p = f(ps). Therefore, Eq.
qs in the image I 0,t s ; patches p (q) and ps (qs) have the same (1) is reformulated as:
1
coordinates for the center pixel. {ps, p} constitutes the LR- q  p  f   p s   q s  p s   f   p s   q s  p s   q s  p s  (2)
HR training patch pairs and {qs, q} constitutes the LR-HR 2
testing patch pairs. Intuitively, to obtain HR patch q , the derivatives of
I1 
1

1
I1,s mapping function should be firstly obtained. To achieve this,
( H ) we use the algorithm proposed in our previous work that
exploits the multiscale non-local similarity property to select
f
suitable exemplar LR-HR training image patches and
achieves an accurate second-order derivative estimation for
the mapping function. More specifically, patches (colored in
 
I1,s
0 green in Fig. 1(a)) similar to ps and their corresponding HR
I1 
0

1
( I0 ) version (colored in pink in Fig. 1(a)) are used to estimate the
q
f qs
mapping function f that associates ps to p. For more
technical details, the interested reader may refer to [11].

Unified interpolation and denoising

I 0,s
0

Input I p ps
( I 00 )
f

I 1,s
0
 0
I 1
(a) (b) (c) (d)

Fig. 2. Controlled experiments at different settings of p, f’(ps), f’’(ps), and

I 2  0
0
I 2,s qs.. The input LR image was contaminated with Gaussian noise (σ = 10). (a)
LR input image (enlarged using nearest-neighbor interpolation for better
comparison). (b)-(d) are SR results using the second-order-based SLSR
Mapping function f estimation
framework, where (b) p, f’(ps), f’’(ps), and qs. are both noisy, (c) p, f’(ps),
f’’(ps) is noise-free and qs is noisy, (d) p, f’(ps), f’’(ps) are noisy and qs is
(a)
I 0
iter 
noise-free.

f
I (0iter 1) Mapping function Unified interpolation
Image
Final
Output
However, the second-order-based SLSR only works
estimation and denoising
resize H well on noise-free images. Several researchers [8,11] have
pointed out that directly applying SLSR to noisy LR image
(b) would cause noise amplification in the reconstructed HR
Fig. 1. An SLSR flowchart. (a) SR reconstruction inside each iteration and image, as shown in Fig. 2(b). Analytically, if the input image
(b) iteration process. Note that patches p (q) and ps (qs) have the same is noisy, these four terms in Eq. (2) would also be corrupted:
ordinate. p, f’(ps), f’’(ps), and qs. To investigate the term that is the
most essential factor for noise amplification in SLSR, two
2.1 Second-order derivative estimation based SLSR controlled experiments were carried out. In the first
framework empirical experiment, we assume that the ground truth of p,
f’(ps), f’’(ps) are known, which is estimated from the original
SLSR assumes that mapping function f associates each LR- noise-free LR image, whereas qs is calculated from the noisy
HR patch pair {qs, q}as q = f(qs). By applying an Nth order LR input. In the second empirical experiment, we assume
Taylor series, the unspecified f can be locally expanded in ps, that the interpolated image is calculated from the original
if ps and qs are similar in terms of the geometric layout: noise-free image, and other three terms are estimated from
the noisy input. Figs. 2(c) and 2(d) represent the
corresponding reconstruction results under these

2527
Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY TAXILA. Downloaded on May 17,2023 at 06:35:58 UTC from IEEE Xplore. Restrictions apply.
 assumptions. Obviously, we can observe from Fig. 2 that
among these four terms, the noise in qs is more influential
than other three in noise amplification. Therefore, how to
remove the noise in qs, or in other words, the interpolated
version I1,s in SLSR, is fundamental for a noise-robust
SLSR algorithm.
2.2 Simultaneous interpolation and image denoising
Generally, there are two different ways in getting a clean I1,s
from the noisy LR input: performing denoising and
interpolation (bicubic in our method) separately or
simultaneously. Our pilot experiments on BSD500 database
reveal that the former way would weaken fine image details
(see Fig. 3 for details). Therefore, we propose a joint
interpolation and denoising procedure.
(a)
(b) (c) (d) (e)
Fig. 3. Example of our pilot study on the order of denoising and
interpolation. (a) LR image with σ = 10, (b)-(d) are results of our
proposed framework using (b) simultaneous interpolation and denoising,
(c) interpolation+denoising, (d) denoising+interpolation. (e) Original HR
image.
Observed that the noise level becomes lower along
with the decreased image scale in the multiscale image
pyramid [12], this multiscale noise reduction prior is
employed in our interpolation method. Considering that in a
typical image interpolation scenario, a target pixel is
estimated by a weighted averaging of its neighboring pixels,
in which the interpolation weight is determined by the
spatial distance between the target pixel and its neighboring
pixel. Thus, in order to achieve a noise-free interpolation,
we propose to specially select the neighboring pixels with
lower noise level from a multiscale directional pyramid.
I1
I0
P0
y
I-1 P-1
I-2
P-2
I-3
P-3
P-4
I-4
… P-14
I-14
…
P-20
I-20
Fig. 4. The diagram of the proposed joint denoising and interpolation.
Patches Pi i = 0,-1,…,-20 have the same relative coordinate. Note that only
the diagonal pyramid is shown for display convenience. The green line
2528
Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY TAXILA. Downloaded on May 17,2023 at 06:35:58 UTC from IEEE Xplore. Restrictions apply.
represent neighboring pixels used in conventional interpolation and the
pink line show the “best” neighboring pixels used in the proposed method
In our method, three types of directional multiscale
pyramids are exploited. More specifically, they are
generated by blurring and downsampling I0 in horizontal,
vertical, and diagonal directions. Corresponding scaling
factors of d n , n  1, 2, , 20 are used. For pyramids in the
horizontal and vertical directions, variance of the 5×5
Gaussian blur kernel in each scale was set to  n2  d n n ,
where σ denotes the standard deviation of noise in an input
LR image. For the diagonal pyramid, the variance of the
5×5 Gaussian blur kernel at each scale was set
to  n2  d n B2 n . Then, we empirically look for “best”
candidate of each P0 from these directional pyramids. As
shown in Fig. 4, the candidates are located at the same
relative image coordinates to P0 in three directional pyramid.
For instance, if (j, k) are the coordinates of P0’s centered
pixel, its candidate in the nth down-scale is located near the
coordinates (|j×dn|, |k×dn|), n = 1, 2, …, 20. In this way, we
are able to form a total of 60 (3×20) candidate patches for
each noisy patch.
Intuitively, the best candidate for P0 should account for
structure preservation of the noisy input and noise-level
reduction. In this paper, these two factors are associated via
the following objective function:
1 1
min dist  P0 , P- n   min 2 P0  P- n 2   2 d n 1
2
(3)
P- n P n N 2
where P0 denotes the noisy input patch in I0 and P-n is the
in-scale candidate patch in the n-th down-scale image (both
P0 and P-n are normalized to [0, 1]), N represents the pixel
number within P0. The first term in Eq. (3) represents the
data fidelity and the second term approximates the noise
correlation between an original noisy patch and the
multiscale candidate.
After selecting the optimal candidate, it is then used as
the neighboring pixels for estimating the target pixel y in the
interpolated I1,s, as shown in Fig. 4. In a similar fashion, all
pixels in image I1,s are estimated in a raster-scan order and
the noise-free interpolated image I1,s is consequently
obtained. By substituting the joint interpolation and
denoising procedure with simple interpolation of the SLSR
framework in 2.1, we can obtain the noise-robust SLSR
method.
3. EXPERIMENT
In order to evaluate the robustness of the proposed method,
several experiments were conducted on noisy images with
different noise levels. 26 typical images are used as the
ground truth for synthesis experiments. Several state-of-the-
art SR approaches including the Kim’s method [4], deep-
learning-based SR (DLB-SR) [13], the A+ method [14],
MTR method [15], and the NCSR method [9] were selected
 TABLE I Comparisons of average PSNR values [dB] (the first row) and SSIM values (the second row)
for images with different noise levels
Kim+ A++
σ Kim A+ DLB-SR
denoise denoise
25.86 27.59 25.95 27.49
10
0.66 0.77 0.66 0.77
23.65 26.45 23.77 26.49
15
0.54 0.74 0.55 0.74
as comparison baselines. The latter two are noise-robust
methods. Note that Kim’s method, DLB-SR and the A+
method cannot handle noisy images directly. An advanced
image denoising method PGPD [16] is leveraged to further
process their SR results. LR images used in our experiments
are generated by down-sampling (bicubic interpolation) the
original images, followed by corruption of white Gaussian
noise with zero mean and standard deviation σ.
In the proposed method, the patch size was set to 5×5,
with 4 pixels overlapping between adjacent patches. A
standard deviation σ equals to the noise level deviation was
assigned to the Gaussian filters used to generate the
multiscale directional pyramids. In this paper, the noise
level was estimated by the method advocated in [17]. In
addition, the gradual upscaling factor was set to 1.5, and the
down-scaling factor d was set to 0.9.
Table I summarizes the average PSNR and SSIM [18]
values for the abovementioned SR algorithms on LR images
with varying noise levels (σ = 10, and 15). We can see that
the PSNR and SSIM scores in Kim’s method, DLB-SR and
the A+ method are lower than the noise-robust methods.
This is because these three methods lack a noise attenuation
technique and noise amplitude is magnified during
superresolving. Moreover, we observe that after the
denoising step, the PNSR and SSIM values in these three
methods are increased above that of the MTR and NCSR
methods. This confirms that the noise issue in SR cannot be
neglected. More specifically, as the noise level increases
(a) (b) (c) (g)
(h) (i) (j) (k)
Fig. 5. A comparison of the SR results for the noisy image FLower with
σ = 15. (a) A+. (b) Denoised A+ result. (c) NCSR. (g) DLB-SR. (h)
Denoised DLB-SR. (i) The proposed method. (j) The LR image. (k) The
ground truth. Zoom for better comparison
Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY TAXILA. Downloaded on May 17,2023 at 06:35:58 UTC from IEEE Xplore. Restrictions apply.
DLB-SR+
MTR NCNR Ours
denoise
25.84 27.61 24.64 26.55 27.72
0.65 0.79 0.72 0.75 0.82
23.57 26.50 24.27 25.23 26.65
0.53 0.73 0.68 0.65 0.77
from σ = 10 to σ = 15, the gap in PSNR values between
NCSR and the denoised version of A+ is also increased.
This is partly due to images which are seriously
contaminated, the noise-robust method NCSR applies as an
averaging technique and cannot effectively remove noise.
On the contrary, the proposed method consistently
outperforms other SR methods as well as the two-step
framework across all test images. This serves to validate the
effectives of the proposed method in processing of noisy
images.
For quantitative comparison, the 2× SR results for noisy
image Flower with a noise standard deviation of 15 are
shown in Figs. 5. For better visual comparison, we also
provide local magnification in the blue rectangle in the
lower-right corner in each example image. It is observed
that during the SR process, additive noise in the LR image
is also introduced to the output image. Although a post
denoising step is able to remove most noise components, the
magnitude of remnant noise cannot be neglected. As can be
seen from these images, the proposed method not only
provides a better noise removal than any of the compared
algorithms, but also produces fewer jagged or blurry
artifacts.
4. CONCLUSION
In this paper, we presented a new SLSR method for
processing noisy images. The proposed method first
explored the cause of noise magnification in the SLSR
framework and then developed a simultaneous
interpolation and denoising scheme by using the noise
reduction property of multiscale image pyramid.
Substituting conventional interpolation method with this
scheme in the SLSR framework, a noise-robust SLSR is
developed. Extensive experimental results obtained from
different noisy images have validated the effectiveness of
the proposed SR method for detail preservation and noise
removal. The proposed technique produced an improved
super-resolution reconstruction both quantitatively and
perceptually in comparison with other advanced baselines.
2529
 REFERENCES
[1] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-
based super-resolution,” IEEE Comput. Graph. Appl. 22(2),
56-65 (2002).
[2] J. Yu et al., “A unified learning framework for single image
super-resolution,” IEEE Trans. Neural Netw. Learn. Syst.
to appear (2013).
[3] X. Gao et al., “Joint learning for single-image super-
resolution via a coupled constraint,” IEEE Trans. Image
Process. 21(2), 469-480 (2012).
[4] K. Kim and Y. Kwon, “Single-image super-resolution
using sparse regression and natural image prior,” IEEE
Trans. Pattern Anal. Mach. Intell. 32(6), 1127-1133 (2010).
[5] G. Freedman and R. Fattal, “Image and video upscaling
from local self-examples,” ACM Trans. Graph. 30(2)
(2011).
[6] M. Zontak and M. Irani, “Internal statistics of a single
natural image,” in Proc. IEEE Comput. Soc. Conf. on
Computer Vision and Pattern Recognition, pp. 977-984
(2011).
[7] K. Zhang, X. Gao, D. Tao, and X. Li, “Single image super-
resolution with multiscale similarity learning,” IEEE Trans.
Image Process. 24(10), 1648-1659 (2013).
[8] J. Yang, Z. Lin, and S. Cohen, “Fast Image super-
resolution based on in-place example regression,” in Proc.
IEEE Comput. Soc. Conf. on Computer Vision and Pattern
Recognition, pp. 1059-1066, (2013).
[9] W. Dong, et al, “ Nonlocally centralized sparse
representation for image restoration,” IEEE Transactions
on Image Processing, 22(4), 1620-1630, (2013).
2530
Authorized licensed use limited to: UNIV OF ENGINEERING AND TECHNOLOGY TAXILA. Downloaded on May 17,2023 at 06:35:58 UTC from IEEE Xplore. Restrictions apply.
[10] A. Singh, F. Porikli, and N. Ahuja, “Super-resolving noisy
images,” in Proc. IEEE Comput. Soc. Conf. on Computer
Vision and Pattern Recognition, pp. 2846-2853, (2014).
[11] J. Hu and Y. Luo, “Single-image superresolution based on
local regression and nonlocal self-similarity,” Journal of
Electronic Imaging, 23(3), 033014-033014-14, (2014).
[12] M. Zontak and M. Irani, “Internal statistics of a single
natural image,” in Proc. IEEE Comput. Soc. Conf. on
Computer Vision and Pattern Recognition, pp. 977-984
(2011).
[13] Z. Wang, et al. "Deep networks for image super-resolution
with sparse prior." in ICCV, pp. 370-378, (2015).
[14] R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted
Anchored neighborhood regression for fast super-
resolution,” In Asian Conference on Computer Vision
(ACCV 2014), pp.1-5, (2014).
[15] E. Lopez-Rubio, “Super-resolution from a single noisy
image by the median filter transform,” SIAM Journal on
Imaging Sciences, 9(1), 82-115 (2016).
[16] J. Xu, et al., “Patch group based nonlocal self-similarity
prior learning for image denoising,” in Proc. of IEEE Int.
Con. on Computer Vision, pp. 244-252, (2015).
[17] X. Liu, M. Tanaka and M. Okutomi, “ Noise level
estimation using weak textured patches of a single noisy
image,” in Proc. of IEEE Int. Con. on Image Processing, pp.
665-668, (2012).
[18] Z. Wang et al., “Image quality assessment: from error
visibility to structural similarity,” IEEE Trans. Image
Process. 13(4), 600-612 (2004).