Chapter-1

INTRODUCTION
1.1. BASICS OF IMAGE PROCESSING
An image is a two-dimensional picture, which has a similar appearance to
some subject usually a physical object or a person. Image is a two-dimensional, such
as a photograph, screen display, and as well as a three-dimensional, such as a statue.
They may be captured by optical devices such as cameras, mirrors, lenses, telescopes,
microscopes, etc. and natural objects and phenomena, such as the human eye or water
surfaces.
1.1.1. Image
In simple words an image is a representation of a real scene, either in black
and white or in color. Technically an image may be defined as a two dimensional
function f(x,y) where x and y are spatial (plane) coordinates (x,y) is called the
intensity or gray level of the image at that point.
1.1.2. Types of Images
There are several standard types of images. They are
1) Binary
2) Gray Scale
3) Color
Binary Image
Binary images are the simplest type of images and can take on two values,
typically black and white, or 0 and 1. A binary image is referred to as a 1-bit image
because it takes only 1 binary digit to represent each pixel. These types of images are
frequently used in applications where the only information required is general shape
or outline.
Binary images are often created from the gray-scale images via a threshold
operation, where every pixel above the threshold value is turned white (1), and those
below it turned black (0). In fig. 1.1 a binary image is shown.
1

Fig. 1.1: Example of Binary Image

Gray-Scale Image
Gray-Scale images are referred to monochrome (one-color) images. They
contain gray-level information, no color information. The number of bits used for
each pixel determines the number of different gray levels available. The typical grayscale image contains 8bits/pixel data, which allows us to have 256 different gray
levels. Fig. 1.2 shows a gray-scale image.

Fig. 1.2: Example of Gray Scale Image

In applications like medical imaging and astronomy, 12 or 16 bits/pixel
images are used. These extra gray levels become useful when a small section of the
image is made much larger to discern details.
Color Image
Color images can be modeled as three-band monochrome image data, where
each band of data corresponds to a different color. The actual information stored in
the digital image data is the gray-level information in spectral band.
2

Typical color images are represented as red, green, and blue (RGB images).
Using the 8-bits monochrome standard as a model, the corresponding color image
would have 24-bits/pixels (8-bits for each of the three color bands red, green, and
blue). Fig. 1.3 shows the RGB image.

Fig. 1.3: Example of RGB Image

1.2. IMAGE PROCESSING

Image processing is any form of signal processing for which the input is an
image, such as photographs or frames of video; the output of image processing can be
either an image or a set of characteristics or parameters related to the image.
INPUT

IMAGE

OUTPUT

(IMAGE)

PROCESSING

(IMAGE/DATA)

Fig. 1.4: Block diagram of image processing

1.2.1. Types of Image Processing
1) Analog Image Processing
2) Digital Image Processing
Analog Image Processing: Analog image processing is any image processing task
conducted on two-dimensional analog signals by analog means (as opposed to digital
image processing).
Digital Image Processing: Digital image processing is the use of computer
algorithms to perform image processing on digital images.

1.2.2. Levels in Digital Image Processing

In digital Image Processing there are three types of computerized processes.
They are:
a) Low-level process
b) Mid-level process
c) High-level process
a) Low-level Process: This process involves primitive operations such as image preprocessing to reduce noise, contrast enhancement, and image sharpening. In this
process both the inputs and outputs are images.
b) Mid-level Process: This process involves tasks such as Image Segmentation
(Partitioning an image into regions or objects), description of those objects to reduce
them to a form suitable for computer processing, and classification (recognition) of
individual objects. In this process the inputs generally are images where as the outputs
are attributes extracted from those images such as edges, contours etc.
c) High-level Process: This process involves making sense of an ensemble of
recognized objects as in image analysis and perform functions normally associated
with vision.
1.2.3. Advantages of Digital Image Processing
a) The digital image has more color and better contrast to it than the analog
image.
b) The digital image is the sharper image.
c) The storage of image in digital form is easier compared to the storage of the
analog; this implies that the memory required for digital images is less.

1.2.4. Fundamental Steps in Image Processing

Fig. 1.5: Steps in image processing

a) Image Restoration: This area deals with improving the appearance of an
image. Image restoration is based on mathematical and probabilistic models of
image degradation. Image Enhancement on the other hand, is based on human
subjective preferences.
b) Color Image Processing: This area has been gaining in importance because
of the significant increase in the use of digital images over the Internet.
c) Wavelets: These are the foundation for representing images in various degrees
of resolution.
d) Compression: This deals with techniques for reducing the storage required
saving an image or the bandwidth required transmitting it.
e) Morphological Processing: This deals with tools for extracting image
components that are useful in the representation and description of shape.
f) Segmentation: Segmentation procedures partition an image into its
constituent parts or objects. In general, the more accurate the segmentation,
the more likely recognition is to succeed.
g) Representation and description: This almost always follows the output of a
segmentation stage, which usually is raw pixel data, constituting either the
boundary of a region or all the points in the region itself.
5

1.2.5. Neighbourhood
A pixel P at coordinates (x,y) has four horizontal and vertical neighbours
whose coordinates are given by (x+1, y), (x-1, y), (x, y+1), (x, y-1).This set of pixels
called the 4-neighbors of P is denoted by N4(P).Each pixel is a unit distance from
(x,y), and some of the neighbours of P lie outside the digital image if (x,y) is on the
border of the image. The four diagonal neighbours of P have coordinates (x+1, y+1),
(x+1, y-1), (x-1, y+1), (x-1, y-1)are denoted by Ng (P).
NW

SW

NE

SE

Fig. 1.6: Neighbourhoods

1.3. Background
The term inpainting comes from art restoration, where it is also called retouching. The term inpainting is borrowed from paper art, where restoration artists are
tasked with restoring faded and damaged paintings. Medieval artwork started to be
restored as early as the Renaissance, the motives being often as much to bring
medieval pictures up to date" as to fill-in any gaps. The need to retouch the image in
an unobtrusive way extended naturally from paintings to photography and film. The
purposes remained the same: to revert deterioration (e.g. scratches and dust spots), or
to add or remove elements. In art however, the major concern is to hide the damage in
whichever way complements the existing pigments and image the best, rather than
repaint the damage parts of the painting since erasing paintings is generally not an
option
Image retouching ranges from the restoration of paintings to scratched
photographs or films to the removal or replacement of arbitrary objects in images.
6

Retouching can furthermore be used to create special effects (e.g., in movies).

Ultimately, retouching should be carried out in such a way that when viewing the endresult it is impossible for an arbitrary observer, or at least very hard, to determine that
the image has been manipulated or altered.
In the digital domain, the inpainting problem first appeared under the name
error concealment" in telecommunications, where the need was to fill-in image
blocks that had been lost during data transmission. One of the first works to address
automatic inpainting is known as image dis-occlusion," since it treated the image gap
as an occluding object that had to be removed, and the image underneath would be the
restoration result. Popular terms used to denote inpainting algorithms are also image
completion" and image fill-in

1.4.

Digital Inpainting

Digital Inpainting alludes to how to perform inpainting digitally through image

processing in some sense. Thereby also automating the process and reducing the

Fig 1.7:.Example of manual inpainting performed by a artist.

interaction required by the user. Ultimately, the only interaction required by the user
is the selection of the region of the image to be Inpainted.
Inpainting is the process of reconstructing lost or deteriorated parts of an
image. The process of inpainting utilizes the background information to fill the
missing or target region of image. Initially inpainting is used for scratch removal. The
other applications include object removal, text removal and other automatic
modification of images.
7

The idea of object removal is to remove objects from digital photographs and
fill the hole with the information extracted from the surrounding area. In this work,
Super-resolution based Inpainting is used for object removal in Digital photographs.
Inpainting vs Denoising
Image inpainting is different than image denoising. Image inpainting is an
iterative method for repairing damaged pictures or removing unnecessary elements
from pictures. Classical image denoising algorithms don't apply to image inpainting.
In common image enhancement applications, the pixels contain both information
about real data and the noise, while in image inpainting, there is no significant
information in the region to be inpainted. The information is mainly in the regions
surrounding the areas to be inpainted. Another difference lies within the size of the
data to be processed, the region of missing data in inpainting is usually large like long
cracks in photographs, superimposed large fonts, and so on.

1.5. The Fundamentals of Digital Inpainting

Digital inpainting refers, as already mentioned, to inpainting through some sort of
image processing. The digital inpainting process can be looked upon as a linear or
non-linear transformation as illustrated in Figure 1.2, where i0 is the original image

Image processing
f
Fig 1.8: Linear Transformation through an Image Processor f
and i is the transformed image (i.e., the digitally inpainted image). The image
processor can be looked upon as a function f as follows:
f : io i i.e.

f (io ) i

The following steps describe the general solution to the problem :

STEP 1: SPECIFY
STEP 2: = THE BOUNDARY OF
STEP 3: INITIALIZE

STEP 4: FOR ALL PIXELS X, Y INPAINT X, Y IN BASED ON

INFORMATION IN
The explanation is as follows: Step 1 lets the user specify the region to be inpainted,
Step 2 computes the boundary of the region and Step 3 initializes the region by for
example, clearing existing color information. The for-loop inpaints the region based
on information of its surroundings.

1.6. Aim of Project

The aim of the project is to design a novel frame work for exemplar based
inpainting by combining inpainting and Super-resolution in a two-step approach to
improve the trade-off between quality and complexity.

1.7.

Overview

Although tremendous progress has been made in the past years on inpainting,
difficulties exist when the hole or the area of object to be removed is very large and
the computational time required in general is high. These two problems are addressed
by considering a two-step or hierarchical approach in which inpainting is performed
on a input image and a super resolution algorithm is used to construct a high
resolution (HR) image. Super-resolution (SR) imaging aims to overcome or
compensate the limitation or shortcomings of the image acquisition device/system
and/or possibly ill-posed acquisition conditions to produce a higher-resolution image
based on a set of images that were acquired from the same scene. With rapid progress
in image processing for visual communications and scene understanding, there is a
strong demand for providing the viewer with high-resolution imaging not only for
providing better visualization (fidelity issue) but also for extracting additional
information details (recognition issue). A HR image makes it easy to achieve a better
classification of regions in a multi-spectral remote sensing image or to assist
radiologist for making diagnosis based on a medical imagery.

Super-resolution refers to creating an enhanced high resolution (HR) image

from one or more low resolution (LR) images [7]. The approach used in this project is
based on regularization frame work. Another class of super resolution methods
generates an HR image from a single LR image or a frame [7]. These methods are
9

referred to as Example-based SR or image hallucination. In example based SR,

correspondences between HR and LR patches are learned [6] from a group of HR-LR
patches known as Dictionary and then applied to a low resolution image to recover its
higher resolution version.
In this work, a non-linear regularization method based on multiscale
morphology is used for edge preserving SR reconstruction [8]. In this method wSuper
Resolution image reconstruction is considered as a deblurring problem and the inverse
problem is solved using Bregman iterations. The HR image is estimated based on
some prior knowledge about the image in the form of regularization. A new
regularization method based on multiscale morphological filters is proposed.
Morphological operators are used for extracting structures from images. Image
segmentation, image denoising and image fusion can be done successfully using
morphological operations. Better results can be obtained by combining bregman
iteration.

1.8.

Organization of Thesis
Remaining portion of this work covers the following: Chapter 2 describes

about different existing inpainting methods, Chapter 3 gives detailed description

about super-resolution and about various super-resolution methods. Chapter 4 outlines
frame work of proposed method. Chapter 5 describes in detail the algorithm for
Exemplar-based inpainting. Chapter 6 explains the algorithm for Super-resolution
using Bregman iteration. A brief description of steps involved in Example based
Super-resolution are described in Chapter 7. Results obtained using proposed methods
are shown in Chapter 8. Finally, concluding remarks and future are drawn in
Chapter 5.

1.9.

Resources Required

Matlab 7.11 (R 2010b)

Adobe Photoshop CS4

10

Chapter-2
LITERATURE SURVEY
2.1. Types of Inpainting
Mainly three groups of image inpainting algorithms can be found. The first
one to be noted is structural inpainting, the second one is texture inpainting and the
last one is a combination of these two techniques. All these inpainting methods have
one thing in common i.e. they use the information of the known or undestroyed image
areas in order to fill the gap.
Structural inpainting
Structural inpainting uses geometric approaches for filling in the missing information
in the region which should be inpainted. These algorithms focus on the consistency of
the geometric structure.
Textural inpainting
Like everything else the structural inpainting methods have both, advantages and
disadvantages. The main problem is that all the structural inpainting methods are not
able to restore texture. Texture has a repetitive pattern which means that a missing
portion cannot be restored by continuing the level lines into the gap.
Combined Structural and Textural inpainting
Combined structural and textural inpainting approaches simultaneously try to perform
texture and structure filling in regions of missing image information. Most parts of an
image consist of texture and structure. The boundaries between image regions
accumulate structural information which is a complex phenomenon. This is the result
when blending different textures together. That is why, the state of the art inpainting
method attempts to combine structural and textural inpainting.

2.2. RELATED WORK:

2.2.1.Diffusion based Inpainting
Diffusion based Inpainting was the first digital Inpainting approach. In this approach
missing region is filled by diffusing the image information from the known region
11

into the missing region at the pixel level. Basically these algorithms are based on
theory of variational method and Partial Differential equation (PDE). The diffusionbased Inpainting algorithm produces superb results or filling the non-textured or
relatively smaller missing region. The drawback of the diffusion process is it
introduces some blur, which becomes noticeable when filling larger regions. All the
PDE based in painting models are more suitable for completing small, non-textured
target region.
2.2.2.Texture Synthesis Based Inpainting
Texture synthesis based algorithms are one of the earliest methods of image
Inpainting. And these algorithms are used to complete the missing regions using
similar neighbourhoods of the damaged pixels. The texture synthesis algorithms
synthesize the new image pixels from an initial seed. And then strives to preserve the
local structure of the image . All the earlier Inpainting techniques utilized these
methods to fill the missing region by sampling and copying pixels from the
neighbouring area. For e. g, Markov Random Field (MRF) is used to model the local
distribution of the pixel. And new texture is synthesized by querying existing texture
and finding all similar neighbourhoods. Their differences exist mainly in how
continuity is maintained between existing pixels and Inpainting hole. The main
objective of texture synthesis based inpainting is to generate texture patterns, which is
similar to a given sample pattern, in such a way that the reproduced texture retains the
statistical properties of its root texture .
2.2.3. PDE based Inpainting
This algorithm is the iterative algorithm. The main idea behind this algorithm is to
continue geometric and photometric information that arrives at the border of the
occluded area into area itself . This is done by propagating the information in the
direction of minimal change using isophote lines. This algorithm will produce good
results if missed regions are small one. But when the missed regions are large this
algorithm will take so long time and it will not produce good results. Then inspired by
this work proposed the Total Variational (TV) Inpainting model. This model uses
Euler-Lagrange equation and anisotropic diffusion based on the strength of the
isophotes. This model performs reasonably well for small regions and noise removal
applications. But the drawback of this method is that this method neither connects
broken edges nor greats texture patterns. These algorithms were focused on
12

maintaining the structure of the Inpainting area. And hence these algorithms produce
blurred resulting image. Another drawback of these algorithms is that the large
textured regions are not well reproduced.

2.2.4. Exemplar based Inpainting.

The exemplar based approach is an important class of inpainting algorithms. And they
have proved to be very effective. Basically it consists of two basic steps: in the first
step priority assignment is done and the second step consists of the selection of the
best matching patch. The exemplar based approach samples the best matching patches
from the known region, whose similarity is measured by certain metrics, and pastes
into the target patches in the missing region. Exemplar- based Inpainting iteratively
synthesizes the unknown region i.e., target region, by the most similar patch in the
source region. According to the filling order, the method fills structures in the missing
regions using spatial information of neighbouring regions. This method is an efficient
approach for reconstructing large target regions.

2.3. SUPER-RESOLUTION
Super-resolution (SR) imaging aims to overcome or compensate the limitation
or shortcomings of the image acquisition device/system and/or possibly ill-posed
acquisition conditions to produce a higher-resolution image based on a set of images
that were acquired from the same scene. With rapid development and deployment of
image processing for visual communications and scene understanding, there is a
strong demand for providing the viewer with high-resolution imaging not only for
providing better visualization (fidelity issue) but also for extracting additional
information details (recognition issue). For examples, a high resolution image is
beneficial to achieve a better classification of regions in a multi-spectral remote
sensing image or to assist radiologist for making diagnosis based on a medical
imagery. In video surveillance systems, higher-resolution video frames are always
welcomed for more accurately identifying the objects and persons of interest.
The most direct approach on obtaining higher-resolution images is to improve
the image acquisition device (e.g., digital camera) by reducing the pixel size on the
sensor (e.g., charge-coupled device). However, there is a limitation in reducing the
sensors pixel size in the sensor technology. When the sensors pixel size becomes too

13

small, the captured image quality will be inevitably degraded. This is due to the fact
that the noise power remains roughly the same, while the signal power decreases
proportional to the sensors pixel size reduction. Furthermore, higher cost is required
to increase the chip size. Owing to the above-mentioned, the SR image processing
becomes a promising alternative. The SR imaging research has grown very rapidly,
after it was first addressed by Tsai and Huang in 1984. In view of this, the purpose of
this paper is to provide a comprehensive and updated survey for the SR research
literature and contribute several inspirations for future SR research.
To understand the SR imaging, several fundamental concepts are required to
be clarified. First, it is important to note that an images resolution is fundamentally
different from its physical size. In our context, the objective of SR imaging is to
produce an image with a clearer content from its low-resolution counterpart rather
than simply achieving a larger size of image In other words, the main goal and the
first priority of super-resolution imaging is to fuse the contents of multiple input
images in order to produce one output image containing with more clear and detailed
contents. The physical size of the output image (in terms of total number of pixels)
could be the same as any one of the input images or subject to further enlargement
using an image interpolation method. Second, in our context, the term resolution of
super-resolution is referred to the spatial resolution of the image, not the temporal
resolution of the image sequence. The latter is commonly expressed in terms of the
number of frames captured per second (i.e., frame rate). Third, it is worthwhile to note
that the term super-resolution has been used in other research areas as well. For
example, in the field of optics, super-resolution refers to a set of restoration
procedures that seek to recover the information beyond the diffraction limit. In
another example on the scanning antenna research, the super-resolution technique is
exploited to resolve two closely spaced targets when a one-dimensional stepped
scanning antenna is used.
The limitation of SR computation mainly comes from the following factors
interpolation error, quantization error, motion estimation error and optical-blur. Baker
and Kanade showed that, for a sufficiently large resolution enhancement factor, any
smoothness prior image model will result in reconstructions with very little highfrequency content. Lin et al. studied a numerical perturbation model of
reconstruction-based SR algorithms for the case of translational motion and the
learning-based SR algorithms. Robinson and Milanfar analysed this issue using
14

Cramer-Rao bounds. A thorough study of SR performance bounds would understand

the fundamental limits of the SR imaging to find the balance between expensive
optical imaging system hardwares and image reconstruction algorithms.

The objective of SR image reconstruction is to produce an image with a higher

resolution based on one or a set of images captured from the same scene. In general,
the SR image techniques can be classified into four classes: (i) frequency domainbased approach (ii) interpolation-based approach (iii) regularization-based approach
and (iv) learning-based approach. The first three categories get a higher-resolution
image from a set of lower resolution input images, while the last one achieves the
same objective by exploiting the information provided by an image database.

2.3.1. FREQUENCY-DOMAIN-BASED SR IMAGE APPROACH

The first frequency-domain SR method can be credited to Tsai and Huang
where they considered the SR computation for the noise-free low-resolution images.
They proposed to first transform the low-resolution image data into the discrete
Fourier transform (DFT) domain and combined them according to the relationship
between the aliased DFT coefficients of the observed low-resolution images and that
of the unknown high-resolution image. The combined data are then transformed back
to the spatial domain where the new image could have a higher resolution than that of
the input images. Rhee and Kang exploited the Discrete cosine transform (DCT) to
perform fast image de-convolution for SR image computation. Woods et al. presented
an iterative expectation maximization (EM) algorithm for simultaneously performing
the registration, blind de-convolution, and interpolation operations.
The frequency-domain-based SR approaches have a number of advantages.
First, it is an intuitive way to enhance the details (usually the high-frequency
information) of the images by extrapolating the high-frequency information presented
in the low-resolution images. Secondly, these frequency-domain-based SR approaches
have low computational complexity. However, the frequency-domain based SR
methods are insufficient to handle the real-world applications, since they require that
there only exists a global displacement between the observed images and the linear
space-invariant blur during the image acquisition process. Recently, many researchers
have begun to investigate the use of the wavelet transform for addressing the SR
problem to recover the detailed information (usually the high-frequency information)
15

that is lost or degraded during the image acquisition process. This is motivated by that
the wavelet transform provides a powerful and efficient multi-scale representation of
the image for recovering the high-frequency information. These approaches typically
treat the observed low-resolution images as the low-pass filtered sub bands of the
unknown wavelet-transformed high-resolution image. The aim is to estimate the finer
scale sub band coefficients, followed by applying the inverse wavelet transform, to
produce the high-resolution image. To be more specific, take the 2 2 SR
computation as an example. The low-resolution images are viewed as the
representation of wavelet coefficients after several levels (say, N levels) of
decomposition. Then, the high-resolution image can be produced by estimating the (N
+ 1)th scale wavelet coefficients, followed by applying the inverse wavelet
decomposition. In Ei-Khamy et al. proposed to first register multiple low-resolution
images in the wavelet domain, then fuse the registered low-resolution wavelet
coefficients to obtain a single image, followed by performing interpolation to get a
higher-resolution image. Ji and Fermuller proposed a robust wavelet SR approach to
handle the error incurred in both the registration computation and the blur
identification computation.

2.3.2. INTERPOLATION-BASED SR IMAGE APPROACH

The interpolation-based SR approach constructs a high resolution image by
projecting all the acquired low-resolution images to the reference image, then fuses
together all the information available from each image, due to the fact that each lowresolution image provides an amount of additional information about the scene, and
finally deblurs the image. Note that the single image interpolation algorithm cannot
handle the SR problem well, since it cannot produce those high-frequency
components that were lost during the image acquisition process. The quality of the
interpolated image generated by applying any single input image interpolation
algorithm is inherently limited by the amount of data available in the image. The
interpolation-based SR approach usually consists of the following three stages (i) the
registration stage for aligning the low-resolution input images, (ii) the interpolation
stage for producing a higher-resolution image, and (iii) the deblurring stage for
enhancing the reconstructed high-resolution image produced in the step (ii). The
interpolation stage plays a key role in this framework. There are various ways to
perform interpolation. The simplest interpolation algorithm is the nearest neighbour
16

algorithm, where each unknown pixel is assigned with an intensity value that is same
as its neighbouring pixels. But this method tends to produce images with a blocky
appearance. Ur and Gross performed a non-uniform interpolation of a set of spatially
shifted low-resolution images by utilizing the generalized multichannel sampling
theorem. The advantage of this approach is that it has low computational load, which
is thus quite suitable for real-time applications. However, the optimality of the entire
reconstruction process is not guaranteed, since the interpolation errors are not taken
into account. Bose and Ahuja

used the moving least square (MLS) method to

estimate the intensity value at each pixel position of the high-resolution image via a
polynomial approximation using the pixels in a defined neighbourhood of the pixel
position under consideration. Furthermore, the coefficients and the order of the
polynomial approximation are adaptively adjusted for each pixel position. Three steps
that are conducted iteratively. Irani and Peleg proposed an iterative backprojection
(IBP) algorithm, where the high-resolution image is estimated by iteratively
projecting the difference between the observed low-resolution images and the
simulated low resolution images. However, this method might not yield unique
solution due to the ill-posed nature of the SR problem. A projection onto convex sets
(POCS) was proposed by Patti and Tekalp to develop a set-theoretic algorithm to
produce the high-resolution image that is consistent with the information arising from
the observed low-resolution images and the prior image model. These information are
associated with the constraint sets in the solution space; the intersection of these sets
represents the space of permissible solutions. By projecting an initial estimate of the
unknown high-resolution image onto these constraint sets iteratively, a fairly good
solution can be obtained. This kind of method is easy to be implemented; however, it
does not guarantee uniqueness of the solution. Furthermore, the computational cost of
this algorithm is very high.

2.3.3. REGULARIZATION-BASED SR IMAGE APPROACH

Motivated by the fact that the SR computation is, in essence, an ill-posed
inverse problem basic idea of these regularization-based SR approaches is to use the
regularization strategy to incorporate the prior knowledge of the unknown highresolution image. From the Bayesian point of view, the information that can be
extracted from the observations (i.e., the low-resolution images) about the unknown
signal (i.e., the high-resolution image) is contained in the probability distribution of
17

the unknown. Then, the unknown high-resolution image can be estimated via some
statistics of a probability distribution of the unknown high-resolution image, which is
established by applying Bayesian inference to exploit the information provided by
both the observed low-resolution images and the prior knowledge of the unknown
high-resolution image numerous regularization based SR algorithms have been
developed for addressing this issue.

18

Chapter 3
INPAINTING ALGORITHM
The task of object removal becomes complicated when area of object to be
removed or inpainted becomes large. So far a number of methods for inpainting had
been proposed.
In this method, a new framework for inpainting is proposed which is a
combination of low resolution picture inpainting and a super resolution algorithm
based on multiscale morphology.
This process of inpainting is mainly divided into two sequential steps. The
first one is a patch sampling method known as Exemplar based inpainting. The
inpainting is performed on a LR (resized) version of an input image. This is because a
low resolution picture is less contaminated by noise and is composed of main scene
structures. Also, as the picture to inpaint is smaller than the original one, the
computation time is reduced compared to the one necessary to inpaint the original
image. The inpainted version of the image obtained is given as input to the Super
Resolution algorithm. Its goal is to enhance the resolution of the image. Fig. 1 shows
the frame work of the proposed method which is as follows:
1) A low resolution picture is built from original image
2) An exemplar based inpainting algorithm is applied to remove the object.
3) Super Resolution algorithm is applied to the input image.
Original
Image
LR
image

Exemplar
based
Inpainting

Super
Resolution
using
Morphologic
Regularization

Image mask

Fig.3.1.The frame work of the proposed method

19

Output
(Inpainted)
Image

3.1.EXEMPLAR BASED INPAINTING

3.1.1. KEY OBSERVATIONS
Exemplar-based synthesis suffices
The core of our algorithm is an isophote-driven image sampling process. It is
well-understood that exemplar-based approaches perform well for two-dimensional
textures .But, we note in addition that exemplar-based texture synthesis is sufficient
for propagating extended linear image structures, as well; i.e., a separate synthesis
mechanism is not required for handling isophotes.
Figure 3.2 illustrates this point. For ease of comparison, we adopt notation
similar to that used in the inpainting literature. The region to be filled, i.e., the target
region is indicated by , and its contour is denoted by

. The contour evolves

inward as the algorithm progresses, and so we also refer to it as the fill front. The
source region, , which remains fixed throughout the algorithm, provides samples
used in the filling process.
We now focus on a single iteration of the algorithm to show how structure and
textures are adequately handled by exemplar based synthesis. Suppose that the square
template p centred at the point p (fig. 3.2b), is to be filled. The best-match
sample from the source region comes from the patch p , which is most similar to
those parts that are already filled in p . In the example in fig. 3.2b, we see that if p
lies on the continuation of an image edge, the most likely best matches will lie along
the same (or a similarly coloured) edge (e.g., q and q in fig.3.2c).
1

11

All that is required to propagate the isophote inwards is a simple transfer of

the pattern from the best-match source patch). (Fig.3.2d). Notice that isophote
orientation is automatically preserved. In the figure, despite the fact that the original
edge is not orthogonal to the target contour

, the propagated structure has

maintained the same orientation as in the source region. In this work we focus on a
patch-based filling approach (as opposed to pixel-based). Furthermore, we note that
patch based filling improves the accuracy of the propagated structures.

20

Fig.3.2. Structure propagation by exemplar-based texture synthesis.

(a)Original image, with the target region , its contour , and the source
region clearly marked. (b) We want to synthesize the area delimited by the
patch p centred on the point p . (c) The most likely candidate
matches for p lie along the boundary between the two textures in the
source region, e.g., q and q . (d) The best matching patch in the
candidates set has been copied into the position occupied by p , thus
achieving partial filling of . Notice that both texture and structure (the
separating line) have been propagated inside the target region. The target
region has,now, shrank and its front has assumed a different shape.
1

11

Filling order is critical

The previous section has shown how careful exemplar-based filling may be
capable of propagating both texture and structure information. This section
demonstrates that the quality of the output image synthesis is highly influenced by the

21

Onion peel
Desiderata
Fig.3.3. The importance of the filling order when dealing with concave
target regions. (a) A diagram showing an image and a selected target
region (inwhite). The remainder of the image is the source. (b,c,d)
Different stages in the concentric-layer filling of the target region. (d)
The onion-peel approach produces artefacts in the synthesized horizontal
structure. (b,c,d) Filling the target region by an edge-driven filling
order achieves the desired artefact-free reconstruction. (d) The final
edge-driven reconstruction, where the boundary between the two
background image regions has been reconstructed correctly.
order in which the filling process proceeds. Furthermore, we list a number of desired
properties of the ideal filling algorithm. A comparison between the standard
concentric layer filling(onion-peel) and the desired filling behaviour is illustrated in
fig. 3.3. Figures 3.3 b, c, d show the progressive filling of a concave target region via
an anti-clockwise onion-peel strategy. As it can be observed, this ordering of the filled
patches produces the horizontal boundary between the background image regions to
be unexpectedly reconstructed as a curve.
A better filling algorithm would be one that gives higher priority of synthesis
to those regions of the target area which lie on the continuation of image structures, as
shown in figs. 3.3b,c,d. Together with the property of correct propagation of linear
structures, the latter algorithm would also be more robust towards variations in the
22

shape of the target regions. A concentric-layer ordering, coupled with a patch-based

filling may produce further artefacts. Therefore, filling order is crucial to nonparametric texture synthesis. To our knowledge, however, designing a fill order which
explicitly encourages propagation of linear structure (together with texture) has never
been explored, and thus far, the default favourite has been the onion peel strategy.
Another desired property of a good filling algorithm is that of avoiding overshooting artefacts that occur when image edges are allowed to grow indefinitely. The
goal here is finding a good balance between the propagation of structured regions and
that of textured regions (fig. 3.3 b,c,d), without employing two adhoc strategies. As
demonstrated in the next section, the algorithm we propose achieves such a balance
by combining the structure push with a confidence term that tends to reduce sharp
in-shooting appendices in the contour of the target region. As it will be demonstrated,
the filling algorithm proposed in this paper overcomes the issues that characterize the
traditional concentric-layers filling approach and achieves the desired properties of:
(i) correct propagation of linear structures, (ii) robustness to changes in shape of the
target region, (iii) balanced simultaneous structure and texture propagation, all in a
single, efficient algorithm. We now proceed with the details of our algorithm.
3.1.2. Exemplar-based Inpainting Algorithm
Given an input image, we create an image mask for the object to be removed. This
target region to be removed and filled should be selected by the user. The source
region to be used denoted by is defined as original image minus target region .
For texture synthesis, the size of the template window must be specified. Each pixel
maintains a colour value and confidence value, which reflects our confidence in pixel.
The patches in the filling region are given a temporary priority value which should be
calculated in our algorithm. The proposed inpainting i.e. Exemplar based inpainting
method follows the following steps:
1. Filling order computation.
2. Texture synthesis.
3. Updating confidence values.
1)

Filling order computation: In order to find the filling order we need to

calculate the priority of patches in the filling region. The priority of a patch centered

23

on p is calculated using confidence and data terms [2]. The priority is given by the
product of these two terms.
P(p) = C(p) D(p).

(1)

Where C(p) is the confidence term and D(p) is the data term[2].

Fig.3.4.Notation diagram. Given the patch p , np is the normal to the contour

of the target region and I p is the isophote (direction and
intensity) at point p. The entire image is denoted with I.

These are defined as follows:

C (p)

q p (

C (q)

, D (p)

where is a normalization factor,

I p .n p

is the area of

orthogonal to front in the point p and

(2)

denotes orthogonal operator.

During initialization C(p) is set to zero i.e. C(p) = 0

C(p)=1

p , n p is a unit vector

p , and

C(p) is set to 1 i.e.

p I , .

The confidence term is a measure of amount of reliable information surrounding the

pixel p. The patches in which many of the pixels are already filled are considered to
be filled first. For example, patches present at the corners and thin tendrils of target
region are filled first because they provide more reliable information against which to
match. The data term is a function of strength of isophotes hitting the fill front at each
iteration.

24

2)

Propagating information of texture and structure: First, the priorities for

each patch in the filling region are computed. Hence the filling order is determined.
The patch with highest priority p is found and it is filled with the information (data)
from the source region. In traditional inpainting techniques, pixel value information is
propagated via diffusion which leads to smoothness and blurring of image [6].
Propagation of image texture is done by direct sampling of the source region [5]. The
patch

q which

is similar to patch with highest priority

in the source region is

found .

Where

d ( p , q )

arg min

d ( p , q )

(3)

is the similarity distance measured between patches centered at

pixels p and q respectively. Similarity distance is defined as the sum of squared

differences between already filled pixels in these two patches.
The value of each pixel to be filled is copied from its corresponding position

(source exemplar).
3) Updating confidence values: After filling the patch p with new pixel values,
the confidence term C(p) should be updated.

C( p) C( p )p p .

(4)

As filling proceeds, confidence values decay, indicating that we are less sure of the
color values of pixels near the centre of the target region.
The above three steps are repeated in an iterative manner until the
target region is filled with the data from source region. After completion of inpainting
the next step is applying super resolution algorithm on output LR image to obtain a
HR image. Algorithm 1 (Table 3.1) briefly explains the algorithmic steps for
Exemplar based inpainting. The subscript t used in Algorithm 1 indicates the current
iteration.

25

Algorithm 1
Extract the manually selected initial front 0 .
Repeat until done:
1a. Identify the fill front t . If t ;, exit.
1b. Compute priorities P(p) p t
2a. Find the patch p with the maximum priority,
i.e.,

p arg max p t P( p)

2b. Find the exemplar q that minimizes

d ( p , q ) .

2c. Copy image data from q to p p

3. Update C(p) p p

Table 3.1. Algorithm for Exemplar-based inpainting.

3.2. EXAMPLE-BASED SUPER RESOLUTION

3.2.1. Introduction
SR methods refer to the process of creating one enhanced resolution image
from one or multiple input low resolution images. These problems are then referred to
as single or multiple images SR, respectively. In both cases, the problem is of
estimating high frequency details which are missing in the input image(s). Example
based inpainting SR-aided inpainting method falls within the context of single-image
SR.
Methods for SR can be broadly classified into two families of methods:
(i) The classical multi-image Super-resolution, and
26

(ii) Example-Based super-resolution

In the classical multi-image SR a set of low-resolution images of the same scene
are taken (at subpixel misalignments). Each low resolution image imposes a set of
linear constraints on the unknown high resolution intensity values. If enough lowresolution images are available (at subpixel shifts), then the set of equations becomes
determined and can be solved to recover the high-resolution image. Practically,
however, this approach is numerically limited only to small increases in resolution
Example-Based Super-Resolution is also termed image hallucination. In
example-based SR, correspondences between low and high resolution image patches
are learned from a database of low and high resolution image pairs (usually with a
relative scale factor of 2, and then applied to a new low-resolution image to recover
its most likely high-resolution version. Higher SR factors have often been obtained by
repeated applications of this process.
Example-based SR has been shown to exceed the limits of classical SR. However,
unlike classical SR, the high resolution details reconstructed (hallucinated) by
example based SR are not guaranteed to provide the true (unknown) high resolution
details.
In this section we briefly explain the steps involved in Example-based Super
Resolution. Algorithm 3 explains these steps.
3.2.2.Example-based SR algorithm
(Algorithm 2)
1) Dictionary building: It consists of the correspondences between low and
high resolution image patches. The unique constraint is that the high-resolution
patches have to be valid, i.e. entirely composed of known pixels. In this method,
high-resolution and valid patches are evenly extracted from the known parts of the
image. The size of the dictionary is a user-parameter which might inuence
the overall speed/quality trade-off. An array is used to store the spatial
coordinates of HR patches (D H R ). Those of LR patches are simply deduced by
using the decimation factor equal to 2.
2) Filling order of the HR picture: The computation of the lling order is similar
to the computation used in Exemplar based Inpainting Algorithm. The lling
27

process starts with the patch H R having the highest priority and which is
composed of known and unknown parts.
3) For the LR patch corresponding to the HR patch having the highest priority, its
best neighbour in the inpainted images of lower resolution is sought. This
search is performed in the dictionary and within a local neighbourhood. Only
the best candidate is kept. From this LR candidate, a HR patch is simply
deduced. Its pixel values are then copied into the unknown parts of the current
HR patch H R .
After the filling of the current patch, the priority value is propagated and the
aforementioned steps are iterated while there exist unknown.

Fig. 3.5. Flowchart of the super-resolution algorithm. The missing parts of

the red block are filled in by the best candidate stemming either from the
dictionary or from the local neighborhood. The top image represents the
original image with the missing areas whereas the bottom one is the result of
the low-resolution inpainting.
28

3.3. Super-Resolution using Morphological Regularization

Bregman Iteration.

and

It is always desirable to generate a high-resolution (HR) image as it shows more

intricate details. However, available sensors have limitation in respect to their
maximum resolution. Thus our basic goal is to develop an algorithm to enhance the
spatial resolution of images captured by an image sensor with a fixed resolution. This
process is called the super resolution (SR) method and has remained an active
research topic for the last two decades. A number of fundamental assumptions are
made about image formation and quality, which in turn lead to different SR
algorithms. These assumptions include the type of motion, the type of blurring, and
also the type of noise. It is also important whether to produce the very best HR image
possible or an acceptable HR image as quickly as possible. Moreover, SR algorithms
may vary depending on whether only a single low-resolution (LR) image is available
(single frame SR) or multiple LR images are available (multiframe SR). Also, SR
image reconstruction algorithms work either: 1) in the frequency domain or 2) in the
spatial domain. In this paper, we focus only on spatial domain approach for multi
frame SR image reconstruction.
The approach reported in this paper is based on the regularization framework,
where the HR image is estimated based on some prior knowledge about the image
(e.g., degree of smoothness) in the form of regularization. The probability based
approach mentioned earlier is equivalent to the concept of regularization; only in this
case it gives a probabilistic meaning to the regularization expression. Tikhonov
regularization based on bounded variation (L2 norm) is one of the popular
regularization methods for SR reconstruction. It imposes smoothness in reconstructed
image, but at the same time loses some details (e.g., edges) present in the image. The
first successful edge preserving regularization method for denoising and deblurring is
the total variance (TV) (L1 norm) method. Another interesting algorithm, proposed by
Farsiu et al., employs bilateral total variation (BTV) regularization. To achieve further
improvement, Li et al. used a locally adaptive BTV (LABTV) operator for the
regularization. Recently, two other regularization terms were proposed for SR image
reconstruction, namely, nonlocal means regularization and steering kernel
regression. Usually, iterative SR image reconstruction methods based on
differentiable regularization terms with L p norm (1 < p 2) use the gradient descent
approach to obtain optimal solution. A group of solvers, evolved from Bregman
29

iteration, is one of recently developed methods for such non differentiable constraint
optimization problems. Marquina and Osher were first to use the Bregman iteration
for fast SR image reconstruction with TV regularization. However, even though all
these regularization terms for SR image reconstruction lead to a stable solution, their
performance depends on optimization technique as well as regularization term. On the
other hand, based on TV regularization, Marquina and Osher obtained superior result
by employing Bregman iteration. So we envisage that even better results would be
obtained by combining Bregman iteration and a more sophisticated regularization
method that can suppress noise in LR images and ringing artifacts occurred during
capturing the details of the HR image. We propose a new regularization method based
on multiscale morphologic filters which are nonlinear in nature. Morphological
operators and filters are well-known tools that can extract structures from images.
They are used in image denoising, image segmentation and image fusion successfully.
Since proposed morphologic regularization term uses nondifferentiable max and min
operators, we develop an algorithm based on Bregman iterations and the forward
backward operator splitting using subgradients. It is seen that the results produced by
the proposed regularization are less affected by aforementioned noise evolved during
the iterative process.
3.3.1. Problem Formulation:
The observed images of a scene are usually degraded by blurring due to atmospheric
noise or inappropriate camera settings. Downsampling is also an important factor in
degradation of images. HR and LR images can be related as follows:
Yk

DFk H k X

ek ,

1, 2...., K

(5)

where Yk represents column vector of kth LR image of size M, X represents column

vector of HR image of size N and
ek

represents column vector of additive noise. Fk is a geometric warp matrix and H k is

the blurring matrix of size

. D is the downsampling matrix of size

and k

is the index of the LR images. Assuming that H k becomes same for all k, it may be
simply denoted as H.
Yk

DFk HX

ek ,

1, 2...., K

30

(6)

Applying upsampling and reverse shifting, Yk (6) becomes

Y k which

denotes

upsampled and reverse shifted kth LR image. Thus from equation (6), we can write
Yk

ek ,

R k HX

Fk 1DT Yk and ek

where Y k

Ensembling all the available

Y

RHX

(7)
Fk 1DT ek
Y k into

. DT is the upsampling operator matrix.

a single HR grid we obtain Y such that

e.

(8)

L2Error - based Estimation of the SR image

The number of unknown pixels in HR grid in X is very large. Therefore a solution to
obtain X by inversion may not be feasible. Therefore, an estimate of HR image X is
found i.e.,
X

arg min[ RHX Y

X

2
2

(9)

When K< N/M, the SR image reconstruction becomes an ill posed problem, and
therefore, it becomes necessary to impose regularization to obtain a stable solution.

Regularization for the SR Reconstruction Algorithm

Regularization and iterative methods are used in conjunction for the restoration of
noisy degraded images in order to solve an ill posed problem said above. To obtain a
stable solution for above equation, we impose a regularization operator

( X ) on

the

estimated HR image X. Therefore, the SR image reconstruction problem can simply

be formulated as
X

arg min{ ( X ) :
X

RHX

Y 2

(10)

where is a scalar constant depending on the noise variance in the LR images. The
above equation (10) represents a constrained minimization problem. Unconstrained
minimization problem is represented as follows:

31

1
X arg min RHX Y
X 2

2
2

( X )

(11)

where is the regularization parameter that controls emphasis between data error
term and regularization term.
The conventional regularization methods choose ( X ) as the high frequency
energy and minimize its pth form to ensure smoothness. In this paper ( X ) is defined
based on morphological filters that suppress noise [8]. Bright and dark noise can be
removed using morphological opening and closing.
3.3.2. Morphologic Regularization:
Let B be disk of unit size with origin at its center and sB be a disk structuring element
(SE) of size s. Then the morphologic dilation Ds ( X ) of an image X of size m n is
given by
max r( sB )(1) {xr }

max r( sB )( 2) {xr }
Ds ( X )

max

r( sB )(mn) {xr }

(12)

Similarly, the morphologic erosion Es ( X ) is given by

min r( sB )(1) {xr }

min r( sB )( 2 ) {xr }
Es ( X )

min

r( sB )(mn) { xr }

(13)

Morphological opening Os ( X ) and closing Cs ( X ) by structuring element sB is given by

Os ( X ) Ds (E s ( X ))

(14)

Cs ( X ) Es (D s ( X ))

The regularization function based on multiscale morphology is formulated as

S

( X ) s 1t [Cs ( X ) Os ( X )] ; 0< <1

(15)

s 1

32

where 1 is a column vector consisting of all the 1s and is the weighing coefficient.
Therefore, the SR reconstruction problem can be written as
s
X min s 1t [Cs ( X ) Os ( X )] : RHX Y
X s 1

(16)

3.3.3. Subgradient Methods and Bregman Iteration :

In this section we develop an algorithm for Super-Resolution through Bregman
iteration using morphologic regularization.
a)Bregman Iteration:
The minimization problem can be formulated as follows:
min ( X ) : T ( X ) 0

(17)

where and T are convex functions defined over

Rn R .

Bregman iteration proposed for minimization problem above is as follows:

Let X 0 p0 0

(n)
X ( n 1) arg min Bp ( X , X ( n) ( X )
X

( n 1)
( n)

p
p T ( X ( n 1) )

(18)

(n)

Here, Bp denotes Bregman distance related to convex function. The iteration

proposed in (18) can be simplified as follows:
1
X ( n 1) arg min RHX Y
X 2
Y

( n 1)

( n)

(Y RHX

( n 1)

2
2

( X )

(19)

In equation (19) (Y RHX (n) ) is the error in the nth estimation which is added to Y(n)
such that RHX Y becomes zero. The unconstrained optimization (first step in
equation (19) can be explicitly solved using the following two step algorithm.

( X )

U (k 1) X (k) H T RT ( RHX (k) Y )

1
X (k 1) arg min X U ( k 1)
X 2

2
2

(20)

33

b)Proposed iterative SR algorithm:

In this section the final proposed iterative algorithm for Super Resolution of LR
image is defined using Bregman iteration and operator splitting.
The algorithm is as follows:
U ( n 1) X (n) H T RT (RHX( n) Y n
1

X ( n 1) arg min (X) X U ( n 1)

X
2
Y ( n 1) Y ( n) (Y RHX ( n 1) )

(21)

In above equation, substituting subgradients in place of gradients in second step, we

have

( X ) 1
(X U(n1) ) 0.
(X )

(22)

The second step in equation (21) can be modified as follows:

X ( n 1) U ( n 1)

( X )
(X )

(23)
X (n)

Therefore, the final proposed SR image reconstruction algorithm by using Bregman

iteration and operator splitting is as shown in Algorithm1.
Y

is a HR grid which consists of number of unknown pixels. FillUnknown ( Y ) in

Algorithm1 is used to fill these unknown pixels with their corresponding neighbor
pixels. is a predefined value chosen depending on the variance of noise in LR
images.
The regularization function ( X ) consists of opening and closing operations
which in turn uses morphologic operators dilation and erosion respectively. These
operators are non-differentiable. Therefore , we compute the subgradients of the
regularization function as in [8].

34

Algorithm 3
Initialize
Y (0) n 0, Y, X (0) FillUnknown(Y );

While RHX ( n) Y

2
2

U ( n 1) X (n) H T RT (RHX( n) Y n

( X )

( n 1)
( n 1)

X
U

( X ) X (n)

( n 1)
( n)
( n 1)
Y
Y (Y RHX
)

End

Table 3.2. Algorithm for Super resolution using Bregman iteration

35

Chapter 4
Results
The proposed method is applied on different images and compared with the Patch
Match method [4] which is available in Adobe Photoshop.
Fig.4.1 4.9 shows the screen shots captured of MATLAB figures. In each of these
screen shots, the first row consists of the original image (input image) and the mask
image i.e., the image specifying the target region or region of object removal
respectively. The second row consists of inpainted image i.e., obtained after
implementing Algorithm 1.
Fig.4.10 shows the results obtained after applying Super resolution (Algorithm 3)
on different natural images. Column (a) gives the original images, column (b) gives
the image masks (object to be removed) and column (c) gives the SR Inpainted
images.
Fig. 4.11 shows the comparision beween the results obtained through proposed
method and the Patch Match method available in Adobe Photoshop (one of the Stateof-art Methods). Column (a) shows the original images, column (b) shows the image
obtained through Patch Match in Adobe Photoshop and (c) gives the output image
obtained through proposed method. We observe that when the object removal is done
using Patch tool in Adobe Photoshop, the region filled in the results obtained is
blurred and does not look reasonable to human eyes.
Table 4.1 shows the time taken for inpainting and super-resolution for some natural
images. Simulation is performed on a laptop with configuration as follows: Intel i5
processor and 4Gb RAM .Software used for simulation is MATLAB 7.11.0. It is
observed that inpainting process consumes more time.
Table 4.2 gives the comparision between the results obtained through Examle-based
Super Resolution(SR) and results obtained through Super Resolution using Bregman
iteration (Algorithm 3) . The comparision is done by using the quality metric Peak
signal-to-noise ratio (PSNR). PSNR1 gives the PSNR of images obtained through
Example-based SR and PSNR2 gives the PSNR of images obtained through SR using
Bregman iteration.

36

Fig.4.1. Object removal using inpainting on image city

Fig.4.2. Object removal using inpainting on image elephant

37

Fig.4.3. Object removal using inpainting on image island

Fig.4.4. Object removal using inpainting on image farm

38

Fig.4.5. Object removal using inpainting on image ship

Fig.4.6. Object removal using inpainting on image tiger

39

Fig.4.7. Object removal using inpainting on image eagle

Fig.4.8. Object removal using inpainting on image yanam

40

Fig.4.9. Object removal using inpainting on image soldier

41

(a)

(b)

(c)

Fig.4.10. Results of the proposed method. (a) Original Picture, (b) Mask Image
(image specifying the object to be removed), and (c) Output picture after SR process

42

(a)

(b)

(c)

Fig.4.11. Comparison of the proposed method. (a) Original Picture, (b) Patch Match
in Adobe Photoshop, and (c) Proposed method

43

Picture

Resolution

Missing

Inpainting

SR

Total

Area

(sec)

(sec)

(sec)

(%)
Farm

256X256

18

88

40

128

Elephant

320X480

17

90

38

128

Tiger

480X320

28

149

38

187

Soldier

320X480

30

140

38

178

Island

316X416

20

105

37

142

City

400X300

40

40

80

Eagle

188X258

13

70

37

107

Ship

181X241

13

76

38

114

Yanam

240X320

18

24

42

63

Table 4.1. Running time for Inpainting and Super-resolution processes

Picture

PSNR1

PSNR2

(Example-based

(SR using Bregman

SR)

Iteration)

Elephant

16.3

16.8

Farm

15.6

18.43

City

21.8

23.4

Island

14.63

15.39

Yanam

17.6

23.53

Soldier

18.1

18.2

Eagle

18.9

21.6

Ship

17.7

19.2

Tiger

17.1

17.6

Table 4.2. Comparision of PSNR(Peak Signal-to-Noise Ratio) of output images

obtained through Example-based SR and SR using Bregman iteration respectively.
44

Chapter 5
CONCLUSION AND FUTURE SCOPE
5.1. Conclusion
A new approach for object removal using inpainting has been successfully
implemented in this thesis. Object removal is done in a hierarchical manner by
combining inpainting and Super-resolution in a two-step approach. Proposed method
is applied on wide variety of images and the obtained results show the effectiveness of
this method compared to other methods such as Patch Match. The Super Resolution
process used in proposed method is proved to be less complex and effective compared
to previous method such as Example based Super Resolution. The proposed method
achieves the main goal of inpainting i.e., the obtained results look reasonable to the
human eyes.

5.2.Future scope
The process of inpainting has a great scope for future work. One of the
possibilities is to use data base of similar images that represents the same scene for
the purpose of texture synthesis. Inspite of using the input image itself for texture
synthesis, the best suitable matching patch can be caluculated from these images. A
different method of Super Resolution method such as classical multi-image Superresolution in conjuction with proposed inpainting process can be used in order to
obtain better image quality.

45

REFERENCES
[1]

M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, in

Proc. 27th Annu. Conf. Comput. Graph. Interact .Tech.,Jul. 2000, pp. 417424.

[2]

A. Criminisi, P. Prez, and K. Toyama, Region filling and

object removal by

examplar-based image inpainting, IEEE Trans. Image Process., vol. 13, no. 9,
pp. 12001212, Sep. 2004.
[3]

O. Le Meur, J. Gautier, and C. Guillemot, Examplar-based inpainting based on

local geometry, in Proc. 18th IEEE Int. Conf. Image Process., Sep.
2011,pp.34013404.

[4]

C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, Patch- Match: A

randomized correspondence algorithm for structural image editing, ACM
Trans. Graph., vol. 28, no. 3, p. 24, Aug. 2009.

[5]

A. A. Efros and T. K. Leung, Texture synthesis by non-parametric sampling,

in Proc. 7th IEEE Comput. Vis. Pattern Recognit., Sep. 1999, pp. 10331038.

[6]

O. Le Meur and C. Guillemot, Super-resolution-based inpainting, in Proc.

12th Eur. Conf. Comput. Vis., 2012, pp. 554567.

[7]

D. Glasner, S. Bagon, and M. Irani, Super-resolution from a single image, in

Proc. IEEE Int. Conf. Comput. Vis., vol. 10. Oct. 2009, pp. 349356.

[8]

Purkait, P., Chanda, B.: Super resolution image reconstruction through bregman
iteration using morphologic regularization. IEEE Transactions on Image
Processing 21, 40294039 (2012).

[9] A. Marquina and S. J. Osher, Image super-resolution by TVregularization and

Bregman iteration, J. Sci. Comput., vol. 37, no. 3, pp. 367382, Dec. 2008.
[10] N. Komodakis and G. Tziritas, Image completion using efficient belief
propagation via priority scheduling and dynamic pruning, IEEE Trans. Image
Process., vol. 16, no. 11, pp. 26492661, Nov. 2007.
[11] X. Li, Y. Hu, X. Gao, D. Tao, and B. Ning, A multi-frame image superresolution method, Signal Process., vol. 90, no. 2, pp. 405414, Feb.2010.

46

[12]

A. Blake and A. Zisserman, Visual Reconstruction. Cambridge, MA, USA:

MIT Press, 1987.

[13]

Y. Pritch, E. Kav-Venaki, and S. Peleg, Shift-map image editing, in Proc.

Int. Conf. Comput. Vis., Sep. 2009, pp. 151158.

[14]

K. He and J. Sun, Statistics of patch offsets for image completion, inProc.

12th Eur. Conf. Comput. Vis., 2012, pp. 1629.

[15]

J. Hays and A. A. Efros, Scene completion using millions of photographs,

ACM Trans. Graph., vol. 26, no. 3, p. 4, 2007.

[16]

http://www.en.wikipedia.org/wiki/Inpainting

[17]

http://www.math.ucla.edu/~imagers/htmls/inp.html

47