Soft-Edge Assisted

Network for Single Image

Super-Resolution
Presented by,
Parvathy B S
GUIDED BY :Smt Deepa A K Mtech SP S4 09

1
Table of content
 Introduction
 Literature survey
 Objective
 Methodology
 Previous method
 Proposed method
 Software used
 Expected result
 Limitations

2
Table of content
 Conclusion
 Modification
 Work plan
 Reference

3
 Introduction

 Single image super-resolution (SISR) is an extremely hot topic in the field of computer
vision

 To reconstruct a super-resolution (SR) image from a single low-resolution (LR)

 Integrate the image prior knowledge

4
Introduction contin…

 Proposing a soft-edge reconstruction network (Edge-Net)

 The Edge-Net work independently for image soft-edge
 To provide image soft-edge prior for high-quality SR image reconstruction
 Propose an efficient and accurate Soft-edge assisted Network (SeaNet)
 Well-designed network that introduces the Edge-Net to provide image soft-edge prior

5
 Literature survey
Reference Paper
] C. Dong, C. C. Loy, K. He, and X.
Tang, “Learning a deep convolutional
network for image super-resolution[7] (SRCNN), which was the first
J. Kim, J. K. Lee, and K. M. Lee,
“Accurate image super-resolution
using very deep convolutional
networks,[10]
W. Shi et al., “Real-time single image
and video super-resolution using an
efficient sub-pixel convolutional
Content
proposed the Super-Resolution
Convolutional Neural Network
successful model adopting CNN
to the SISR problem and achieved
significant improvement
extended the depth of the
network and addressed the VDSR
with residual learning.
”introduced an efficient sub-pixel
convolutional layer that can learn
an array of upscaling filters to 6
Literature survey contin…
Reference paper
C. Dong, C. C. Loy, and X. Tang,
“Accelerating the super-resolution
convolutional neural network[8]
J. Kim, J. K. Lee, and K. M. Lee,
“Deeply-recursive convolutional
network for image super-
resolution,[11]
W.-S. Lai, J.-B. Huang, N. Ahuja,
and M.-H. Yang, “Deep Laplacian
pyramid networks for fast and
content
adopted a deconvolutional layer at
the end of the model to directly
learn them applying between the
original LR and SR images
investigated a deeply-recursive
convolutional network for SISR,
which introduced recursive-
supervision learning to reduce
model parameters
Proposed a Laplacian pyramid
network to progressively
reconstruct the sub-band residuals 7
 Objective

 To explore a unified framework that can automatically extract and

integrate the soft-edge prior for image super-resolution.

8
Methodology

 Propose a Soft-edge assisted Network (SeaNet) to reconstruct the high-quality SR image

with the help of image soft-edge.

 Proposed SeaNet consists of three sub-nets:

 a rough image reconstruction network (RIRN)
 a soft-edge reconstruction network (Edge-Net)
 an image refinement network (IRN).

9
Architecture of seanet

10
SEANET

 The rough SR image feature maps extraction and the image soft-edge reconstruction in
Stage-I is represented as

frough=FRIRN (ILR) (1)

fedge=FEdge Net (ILR) (2)

 The output of RIRN and Edge net are merged through a fusion layer

Ffusion=Ffusion([frough,fedge]) (3)

11
 The merged image features are sent to the IRN for high-quality SR image reconstruction
in Stage-II
 ISR=FIRN(ffusion) (4)

 Different from previous works which learn the mapping between LR and HR images
directly
 Propose an edge assisted loss function
 The edge-assisted loss function consists of a content loss and an edge loss.


Therefore, given a training dataset {IiLR,IiHR,IiEdge}Ni=1

(5)

12
 RIRN
(Rough image reconstruction network)
 Rough SR image feature maps reconstruction
 Image low-frequency features can be easily detected.
 Use a 3 × 3 convolutional layer to transform the image from the RGB channel to a
higher dimension
 Five convolutional layers to extract low-frequency image features from the LR Image
 Each layer can be expressed as

 Upsample module

…..(6)
fm=max(0,Wm∗fm-1+bm)

Frough=FUP(f5) …..(7)

13
14
Upsample module

15
 EdgeNet
 Reconstruct image soft-edge from the LR image directly.
 Retain more accurate image edge information
 Use the following curvature formula to obtain the soft-edge

…..(8)
IEdge-=div(ux,uy)

 Where

 x and y represents vertical and horizontal directions

16
Contin…
 Modified version of multi-scale residual network (MSRN)
 Detect the image features at different scales adaptively.
 Mapping between ILR and Iedge
 Reconstructed image soft-edge has the same dimension as the HR image
 Upsampling module in the Edge-Net.

17
Structure of Multiscale residual block

18
Proposed method
 Seanet
 Designed a soft edge reconstruction network (Edge-Net) to reconstruct super resolution
image soft-edge from the LR image directly.
 The image soft-edge can retain more accurate image edge information
 Use the curvature formula to obtain the corresponding soft-edge

19
Implementation details

 Dataset
 DIV2K [42] is a new high-quality dataset for image restoration

 contains 800 training images

 100 validation images and 100 test images.

 Set5 [37], Set14 [38], BSDS100 [39], Urban100 [40], and Manga109 [41] as test
dataset

20
Model settings

 Design three different versions of SeaNet.

 SeaNet (Baseline).
 SeaNet (final)
 SeaNet+.

 The input and output channel of each MSRB and RB are set to 64.

21
Training setting
 First apply the Equation (8) on 800 training images to obtain their corresponding image
soft-edge
 Generate LR images by applying Bicubic interpolation to HR images
 Use the RGB image as input and augment the training data with random horizontal flips
and vertical.
 Implement SeaNet with the PyTorch framework and update it with Adam optimizer.
 The learning rate is initialized as 10−4 and halved every 200 epochs
 Models are implemented with the PyTorch framework
 Trained on NVIDIA Titan Xp GPU.

22
 SeaNet Output

Low resolution input SeaNet output Original Image

23
Training curve

24
Previous method
 SRCNN[7]
 ESPCN[9]
 FSRCNN[8]
 VDSR[10]
 DRCN[11]
 Lap SRN[13]
 DRRN [12]

25
SOFTWARE USED

 PYTHON 3.6
 Using pytorch library

26
LIMITATION

 During training, use Equation (8) to extract soft-edge from images

 Equation (8) is a curvature formula, which is an efficient method but maybe not the best.
 At Stage-II, use a fusion layer to fuse the reconstructed image features and soft-edge provided by
Stage-I.

27
CONCLUSION

 Proposed a new edge-guided image restoration

framework
 Developed an efficient and accurate Softedge assisted Network (SeaNet) for image
super-resolution.
 The Edge-Net is a flexible model

28
Modification

 Edge-Net can be used in many applications

 Image denoising
 Image deblurring

29
WORK PLAN
MONTH Work plan
JAN python software implementation
and language learning, dataset
collection
FEB stage 1 implementation of Seanet
MARCH stage 2 implementation Seanet
APRIL modifications and application
MAY Final output

30
 Reference
 C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional
network for image super-resolution,” in Proc. Eur. Conf. Comput. Vis.(ECCV),
2014, pp. 184–199.[7]
 W. Shi et al., “Real-time single image and video super-resolution using an
efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf.Comput.
Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 1874–1883.[9]
 J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very
deep convolutional networks,” in Proc. IEEE Conf. Comput.Vis. Pattern
Recognit. (CVPR), Jun. 2016, pp. 1646–1654.[10]
 J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for
image super-resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 1637–1645.[11]
 W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep Laplacian pyramid
networks for fast and accurate super-resolution,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 624–632.[13]

31