TECHNOLOGICAL WATCH, DREAM, UPHF, JANUARY 2024 1

Advances in Automatic Image Restoration and

Upscaling: Enhancing Visual Data Fidelity
Cedric Derieux
Université Polytechnique Hauts-de-France, Master 1 Audiovisual Production, DREAM

sophisticated lab for building learning models and conducting

Abstract—The proliferation of digital imagery in various sectors
necessitates advanced methods for improving image quality.
Image restoration and upscaling techniques have become crucial
in transforming degraded or low-resolution images into high-
fidelity, high-resolution outputs. Image upscaling is the process of
increasing the resolution or size of an image, thereby enhancing its
overall visual quality. It is widely used in various fields, including
photography, graphic design, AI art, and video production. The
primary goal of image upscaling is to improve the quality of
images without the need to retake or recreate them, saving time
and resources. Today, with interfaces such as chaiNNer, you can
obtain a high-definition image from a low-definition image, using
a pre-trained upscaling model downloadable from the Internet.
This paper explores the computer science concepts enabling these
techniques, with a focus on the new learning models and
architectures such as SRFormer, HAT, FBCNN, ERSGAN, and
RealESRGAN. We also contrast convolutional neural networks
(CNNs) and transformers, discussing their respective strengths
and limitations in the context of image restoration.
Index Terms— Learning Models, Image Restoration, Upscaling,
Convolutional Neural Network, Transformer-based architectures
I. THE FUNDAMENTALS OF LEARNING MODELS
experiments. Loved for its dynamic nature, PyTorch allows
researchers to modify models on the fly and observe their
performance in real-time. This adaptability proves particularly
useful in the iterative process of designing learning models,
especially when the best approach to achieve optimal results is
uncertain. It is widely used in applications such as computer
vision and natural language processing. [5]
II. CNN-BASED MODELS
Convolutional Neural Network (CNN) based models are a
class of deep learning models primarily used for analyzing
visual data. CNNs are designed to automatically and adaptively
learn spatial hierarchies of features from the input data, which
makes them highly effective for tasks like image classification,
object detection, and image restoration. [6]
The network works using convolutional layers (Fig. 1) that
can focus on low-level features like edges and textures in the
early layers, then aggregate and combine these low-level
features to detect higher-level features (like shapes or objects)
in the later layers.

L earning models are the backbone of modern artificial

intelligence (AI). They are essentially programs that have
the capacity to learn from data. Think of them as students who
improve their knowledge over time with study and experience.
In computer science, these learning models adjust their internal
parameters (which are akin to the knowledge and skills a
student acquires) to make fewer errors in tasks they're designed
to perform, like recognizing objects in a picture, predicting the Fig. 1. Typical CNN architecture. [9]
weather, or, in our case, fixing and enlarging images. Deep
learning, a subset of machine learning, involves neural
A. FBCNN
networks with multiple layers that can learn increasingly
abstract features of the data and extract hierarchical features The Fast and Blind Convolutional Neural Network (FBCNN)
from them, enabling the capture of complex patterns necessary is a model designed to restore images without any prior
for high-quality image reconstruction. [1] knowledge of the degradation processes. It utilizes
convolutional layers, which are fundamental building blocks of
CNNs, to learn from generic patterns in the data. This enables
A. PyTorch for Model Development FBCNN to adaptively restore a wide array of image
PyTorch, an open-source machine learning library, is a corruptions. The model demonstrates robustness in blind image
popular tool among AI developers for its flexibility and restoration scenarios, where the type and degree of image
dynamic computation graph. It provides a platform akin to a corruption are unknown. FBCNN consists of four components:
TECHNOLOGICAL WATCH, DREAM, UPHF, JANUARY 2024
decoupler, QF predictor, flexible controller and image
reconstructor. [10] (Fig. 2)
JPEG is one of the most widely-used image compression
algorithms and formats due to its simplicity and fast
encoding/decoding speeds. However, it is a lossy compression
algorithm and can introduce annoying artifacts. FBCNN is a
flexible (Fig. 3) blind JPEG artifacts removal network for real
JPEG image restoration. [11] (Fig. 4)
Fig. 2. Network Architecture of FBCNN [11]
Fig. 3. Flexibility of FBCNN [11]
(a) JPEG (b) FBCNN Q=30
Fig. 4. Comparison between JPEG source image and FBCNN Output
B. GAN
Generative Adversarial Networks (GANs) are a class of
machine learning frameworks invented by Ian Goodfellow and
his colleagues in 2014. They are designed to generate new data
instances that resemble the training data. GANs consist of two
parts: a generator network, which produces new data instances,
and a discriminator network, which evaluates them for
authenticity. The generator improves its output based on
feedback from the discriminator, creating a dynamic adversarial
process. This process leads to the generation of high-quality
data that closely mimics the distribution of the original dataset.
[13]
C. ERSGAN
The Enhanced Residual Sparse Generative Adversarial
Network (ERSGAN) is a variant of GAN that integrates
residual learning and sparse representations in a GAN
framework. Residual learning helps the model to learn complex
functions by reformulating the layers as learning residual
functions with reference to the layer inputs. Sparse
representations, on the other hand, allow the model to focus on
critical data points for image restoration. This approach
2
enhances the quality of the output for images with intricate
degradation patterns. [14]
D. RealESRGAN
RealESRGAN is an advancement of the ESRGAN
framework for practical application, specifically targeting the
restoration of real-world images. It addresses challenges such
as over-smoothing and the loss of fine details, which were
prevalent in predecessor models. RealESRGAN achieves this
by generating textures that closely resemble those found in the
original high-resolution images. This results in restored images
that retain the naturalness and authenticity of the original
images. (Fig. 5) [16]
Fig. 5. Comparison between low-resolution source image and Real-ERSGAN output
III. TRANSFORMER-BASED ARCHITECTURES
Transformer-based architectures are a type of deep learning
model that have revolutionized the field of natural language
processing. The key innovation of Transformer models is the
self-attention mechanism, which weights the importance of
each part of the input data differently. For example, this allows
the model to consider the context of each word in a sentence,
leading to a better understanding of the input data. [18]
Unlike previous architecture Convolutional Neural Networks
(CNNs), Transformers process input sequences in parallel,
making them highly efficient for training and inference. This is
a significant advantage as it reduces the training time.
A. SRFormer
SRFormer is a novel architecture that leverages the power of
transformer models. Transformers are known for their ability to
learn contextual relationships within data sequences, making
them highly effective for tasks involving sequential data.
In the context of SRFormer, these transformer models are
used to process image data. The architecture employs self-
attention mechanisms, a key component of transformer models,
to distinguish and amplify high-resolution features within
images. This is a significant advancement over traditional
Convolutional Neural Network (CNN)-based methods, which
often struggle with predicting fine details during the upscaling
process. [22] (Fig. 6)
The core of SRFormer is the Permuted Self-Attention (PSA),
a simple, efficient, and effective attention mechanism. It allows
the model to build large range pairwise correlations with even
less computational burden than the original Window Self-
Attention (WSA) of SwinIR. SRFormer is specifically designed
TECHNOLOGICAL WATCH, DREAM, UPHF, JANUARY 2024
for single image super-resolution tasks, and it demonstrates
superior performance compared to state-of-the-art methods
from both Transformer and Convolutional networks. [23]
(a) Source Image (b) 4x Upscale SRFormer
Fig. 6. Comparison between low-resolution source image and SRFormer output
B. HAT
The Holistic Attention Transformer (HAT) is another
transformer-based architecture that makes extensive use of
attention mechanisms. These mechanisms enable the model to
interpret images in a comprehensive manner, considering the
entire image as a whole rather than in isolated segments.
This global perspective is crucial for tasks such as image
restoration, where understanding the interplay between various
image segments is necessary to accurately reconstruct or
enhance image details. By paying attention to the entire image,
HAT can understand the context and dependencies between
different parts of the image, leading to more accurate and
holistic image restoration. [25] (Fig. 7)
(a) Source Image (b) 4x Upscale HAT
Fig. 7. Comparison between low-resolution source image and HAT output
IV. COMPARATIVE ANALYSIS
A. CNNs in Image Restoration
CNNs leverage convolutional filters to process image data.
These filters slide over the image to capture local patterns and
textures. For image restoration, CNNs are effective because
they can learn hierarchical feature representations and perform
well in capturing spatial hierarchies, which is beneficial for
tasks like denoising and super-resolution.
Strengths of CNNs:
- Efficient for spatial data processing.
- Locally connected layers lead to fewer parameters and faster
training.
- Well-suited for capturing local dependencies and textures.
Weaknesses of CNNs:
- Limited receptive field size can hinder capturing global
context (though this can be mitigated by deeper architectures or
dilated convolutions).
- Sometimes struggle with restoring larger-scale structures.
3
B. Transformers in Image Restoration
Transformers, on the other hand, use self-attention
mechanisms to weigh the influence of different parts of the
input data. For image restoration, this means they can capture
long-range dependencies and global context more effectively
than CNNs.
Strengths of Transformers:
- Excel at capturing global context and long-range
dependencies within the image.
- Self-attention allows them to focus on relevant parts of the
image regardless of their spatial position.
- Potentially more flexible in modeling various types of image
degradation.
Weaknesses of Transformers:
- Typically require more computational resources and larger
datasets to train effectively.
- Can be slower to train due to their complexity.
- May include many more parameters than CNNs, increasing
the risk of overfitting on smaller datasets.
For image restoration, Transformers are often used when the
task requires understanding the entire image content, such as in
cases of severe degradation where local information is not
sufficient to reconstruct the original image. However, due to
their computational intensity, hybrid approaches combining
CNNs and transformers are also popular, aiming to leverage the
strengths of both architectures.
C. Limitations and Challenges
These advanced image restoration methods face several
challenges:
- Significant computational resources are necessary, especially
during the training phase.
- Large, diverse datasets of high-quality images are required for
effective training, which can be resource-intensive to compile.
- Artifacts may be introduced by the models, particularly in
cases of severe or atypical degradation.
- Overfitting can occur when models are overly optimized for
training data, reducing their generalizability to new images.
- Transformers, while powerful, may exhibit slower
performance and higher resource consumption compared to
CNNs.
V. TRAINING A DEEP LEARNING MODEL FOR IMAGE
RESTORATION
Training a deep learning model for image restoration
involves several steps, from data preparation to model
evaluation. Here is a high-level overview of the process: [29]
Step 1: Dataset Preparation
Before training begins, a dataset consisting of degraded images
and their high-quality counterparts is required. These image
pairs are used to teach the model the mapping from a low-
quality to a high-quality image.
TECHNOLOGICAL WATCH, DREAM, UPHF, JANUARY 2024
- Degraded Images: These are images that have been
purposefully downgraded through processes such as blurring,
adding noise, compression artifacts, or downsampling.
- High-Quality Images: The dataset must also include the
original or enhanced images that the model aims to reproduce.
Step 2: Model Architecture Selection
Selecting the right model architecture is crucial. This choice
depends on the specific task (denoising, deblurring, super-
resolution) and the available computational resources.
Step 3: Loss Function Design
The loss function measures the difference between the model's
output and the ground truth high-quality image. Common loss
functions include Mean Squared Error (MSE), Structural
Similarity Index (SSIM), and perceptual loss based on features
extracted from pretrained networks.
Step 4: Model Training
During training, the model learns to restore images by adjusting
its parameters to minimize the loss function. This process
involves:
- Forward Pass: The model processes the input degraded
images to generate restored outputs.
- Loss Calculation: The loss between the restored output and
the high-quality target image is calculated.
- Backpropagation: The model's parameters are updated in the
direction that reduces the loss, using optimization algorithms
like stochastic gradient descent (SGD) or Adam.
Step 5: Model Evaluation
After training, the model is evaluated on a separate validation
set to assess its generalization capabilities. Performance metrics
might include Peak Signal-to-Noise Ratio (PSNR), SSIM, and
visual inspection.
VI. CONCLUSION
In conclusion, the landscape of automatic image restoration
and upscaling has witnessed significant advancements with the
advent of CNN and transformer-based models. The selection of
an appropriate architecture, whether SRFormer, HAT, FBCNN,
ERSGAN, or RealESRGAN, depends on the specific image
restoration task. The creation of potent models necessitates
careful architecture choice, rigorous training, and continuous
refinement, facilitated by frameworks such as PyTorch. Despite
their potential, these methods contend with ongoing challenges,
driving continuous research to further advance the field of
image restoration.
REFERENCES
[1] Discover the differences between classical and AI upscaling methods,
their benefits, and applications in enhancing image quality – 13 May 2023
[Online] Available: https://unimatrixz.com/topics/ai-upscaler/upscaling-
methods/
[2] chaiNNer Github Official Code [Online] Available:
https://github.com/chaiNNer-org/chaiNNer
4
[3] PyTorch documentation [Online] Available:
https://pytorch.org/docs/stable/index.html
[4] PyTorch Wikipedia [Online] Available:
https://en.wikipedia.org/wiki/PyTorch
[5] What is PyTorch? (Machine/Deep Learning) – Video by IBM Technology
- 11 oct. 2023 [Online] Available:
https://www.youtube.com/watch?v=fJ40w_2h8kk
[6] Different types of CNN models [Online] Available:
https://iq.opengenus.org/different-types-of-cnn-models/
[7] Convolutional Neural Network: Tout ce qu’il y a à savoir – 25 June 2023
[Online] Available: https://datascientest.com/convolutional-neural-
network
[8] How to Develop Convolutional Neural Network Models for Time Series
Forecasting by Jason Brownlee on August 28, 2020 [Online] Available:
https://machinelearningmastery.com/how-to-develop-convolutional-
neural-network-models-for-time-series-forecasting/
[9] Convolutional neural network Wikipedia [Online] Available:
https://en.wikipedia.org/wiki/Convolutional_neural_network
[10] Towards Flexible Blind JPEG Artifacts Removal - Jiaxi Jiang, Kai
Zhang, Radu Timofte - 29 September 2021 [Online] Available:
https://arxiv.org/abs/2109.14573
[11] Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)
Github Official Code [Online] Available: https://github.com/jiaxi-
jiang/FBCNN
[12] Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)
for Windows Github Official Code [Online] Available:
https://github.com/bycloudai/FBCNN-Windows
[13] What is a Generative Adversarial Network? by Thomas Wood [Online]
Available: https://deepai.org/machine-learning-glossary-and-
terms/generative-adversarial-network
[14] ERSGAN: A Beginner's Guide to Harnessing the Power of Advanced AI
Image Enhancement - Zaid Meccai – 24 July 2023 [Online] Available:
https://blog.segmind.com/how-to-get-started-with-the-ersgan-ai-model/
[15] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure
Synthetic Data Github Official Code [Online] Available:
https://github.com/xinntao/Real-ESRGAN
[16] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure
Synthetic Data [Online] Available: https://arxiv.org/abs/2107.10833
[17] What Is the Transformer Architecture and How Does It Work? [Online]
Available: https://datagen.tech/guides/computer-vision/transformer-
architecture/
[18] What is a transformer model? [Online] Available:
https://www.ibm.com/topics/transformer-model
[19] The Transformer Model by Stefania Cristina on January 6, 2023 [Online]
Available: https://machinelearningmastery.com/the-transformer-model/
[20] Transformer (machine learning model) [Online] Available:
https://en.wikipedia.org/wiki/Transformer_%28machine_learning_model
%29
[21] SRFormer: Permuted Self-Attention for Single Image Super-Resolution
(ICCV2023) [Online] Available: https://github.com/HVision-
NKU/SRFormer
[22] SRFormer: Efficient Yet Powerful Transformer Network for Single Image
Super Resolution - Armin Mehri, Parichehr Behjati, Dario Carpio, Angel
Domingo Sappa - 27 October 2023 [Online] Available:
https://ieeexplore.ieee.org/document/10298198
[23] SRFormer: Permuted Self-Attention for Single Image Super-Resolution -
Yupeng Zhou, Zhen Li, Chun-Le Guo, Song Bai, Ming-Ming Cheng, Qibin
Hou - 17 Mars 2023 [Online] Available https://arxiv.org/abs/2303.09735
[24] Activating More Pixels in Image Super-Resolution Transformer - Xiangyu
Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, Chao Dong – 19 Mars 2023
[Online] Available: https://arxiv.org/abs/2205.04437
[25] HAT: Hybrid Attention Transformer for Image Restoration - Xiangyu
Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao
Zhou, Chao Dong – 11 September 2023 [Online] Available:
https://arxiv.org/abs/2309.05239
[26] HAT Github [Online] Available: https://github.com/XPixelGroup/HAT
[27] A Survey of Deep Learning Approaches to Image Restoration - Jingwen
Su, Boyan Xu, Hujun Yin - 25 February 2022 [Online] Available:
https://research.manchester.ac.uk/en/publications/a-survey-of-deep-
learning-approaches-to-image-restoration
[28] On-Demand Learning for Deep Image Restoration – Ruohan Gao,
KristenGrauman – 2 August 2017 [Online] Available:
https://arxiv.org/pdf/1612.01380
[29] A Comprehensive Review of Deep Learning-Based Real-World Image
Restoration - Lujun Zhai, Yonghui Wang, Suxia Cui, Yu Zhou - 6 March
2023 [Online] Available: https://ieeexplore.ieee.org/document/10056934