ICSP2014 Proceedings
IMAGE INPAINTING WITH STRUCTURE AND TEXTURE PROPAGATION
Voronin Viacheslav, Shraifel Igor, Marchuk Vladimir, Tokareva Svetlana, Sherstobitov Alexander
Dept. of Radio-Electronics Systems, Don State Technical University, Rostov-on-Don, Russian
Federation
voroninslava@gmail.com, shraifel17@mail.ru, marchuk@sssu.ru, tokareva@bk.ru,
sherstobitov.alexander@gmail.com
ABSTRACT
Image inpainting technique uses structural and textural
information to repair or fill missing regions of a picture.
This paper focuses on novel image reconstruction method
based on modified exemplar-based technique. The basic
idea is to find an example (patch) from an image using local
binary patterns, and replacing non-existed (‘lost’) data with
it. We propose to use multiple criteria for a patch similarity
search since often in practice existed exemplar-based
methods produce unsatisfactory results. The proposed
method is tested on various samples of images, with
different geometrical features and compared with the state-
of-the-art image inpainting methods.
Index Terms — Inpainting, reconstruction, texture
analysis, structure propagation, local binary patterns
1. INTRODUCTION
Image inpainting or restoration is an important problem of
image processing. The main purpose of inpainting is to
restore missing («empty») pixels using information from
outside of the damaged domain. It has different applications
from restoration of old photos to removal of unwanted
objects in images. Another application of image inpainting
is in video processing for removal of static images or
auxiliary information, such as channel logos, date, time or
subtitles, which cover some useful information from a
viewer.
Most of image reconstruction methods can be divided
into the following three groups: based on solution of partial
differential equations in partial derivatives (PDE) [1, 2];
based on orthogonal transformations [3-4]; based on texture
synthesis [5-7].
PDE methods, also called as diffusion-based, are
restored missing pixels by a smooth propagation of
information from the neighborhood of the holes by a
diffusion process. These techniques often introduces a blur
in sharp transitions in image and image contours, and need a
priori information to select parameters of the method. They
are suitable for removal of scratches and small defects on
the structure of images, but often fail to restore the texture
image and curved contours.
978-1-4799-2186-7/14/$31.00 ©2014 IEEE
A second group of image inpainting methods uses the
frequency domain processing and sparse representation
model. It should be noted that these methods often tend to
blur textures and structures in a recovery of large areas of
missing pixels. These iterative methods are also highly
computationally expensive.
A third group of image inpainting methods is based on
non-parametric sampling model and texture synthesis.
Criminisi et al. [6] have proposed a patch-based method
based on searching for similar patches and coping them
from the true image. These methods are limited to inpaint
linear structures and often fail to recover curvy boundary
edges.
The main drawbacks of the known methods come from
the fact that the most of them are unable to recover the
curved edges and applicable only for scratches and small
defects removal. It should be also noted that these methods
often blur image in the recovery of large areas with missing
pixels. Most of these methods are computationally very
demanding and in appropriate for implementation on
modern mobile platforms.
In this paper we introduce a novel algorithm for
automatic image inpainting based on a modified exemplar-
based technique. A key idea is to find examples (patches)
from an image using multiple criteria for searching
procedure using several terms, including the Euclidean
metric for pixel brightness and Chi-squared histogram
matching distance for local binary patterns.
2. IMAGE MODEL
A simplified mathematical model of the original image can
be represented as follows:
Yi , j = (1 − M i , j ) ⋅ S i, j + M i , j ⋅ Ri , j , i = 1, I , j = 1, J ,
where S i , j are the true image pixels; [
M = M i, j ] is a binary
mask of the distorted values of pixels (1 - corresponds to the
missing pixels, 0 - corresponds to the true pixels); Ri , j are
missing pixels; I is the number of rows, and J is the
number of columns.
Figure 1a shows the image model, where the region Y
is schematically presented in the form of two sub-regions,
658
representing two types of texture regions, δS is the pixel boundary, which consists of two factors (see figure
boundary of the recovery region, R are missing pixels. 1b):
P (δS ) = С (δS ) ⋅ D (δS ) ,

∑l∈Ψ C (l ) ∇I δ⊥S ⋅ nδS

C (δS ) = δS
, D(δS ) = ,
p ΨδS α
p
where δSi , j is the current pixel on the boundary of true

δS
δS pixels; С (δS ) is the confidence term; D (δS ) is the data
term; ΨδS is the adaptive patch of pixels centered at the
(a) (b) pixel δSi , j ; ΨδS is the number of pixels in the adaptive
Fig. 1. The image model.
patch ΨδS ; ∇I δ⊥S is a vector orthogonal to the gradient at
3. THE PROPOSED ALGORITHM the point δSi , j ; nδS is a vector orthogonal to the boundary
The proposed algorithm is a modification of the example- δS at the point δSi , j ; and α is a normalization factor
based image inpainting method proposed by Criminisi et al. ( α = 255 for a typical grey-level image)).
[6]. The exemplar-based method (EBM) uses a priority
function to select the best patch and copy it to fill-in regions.
Such an approach allows propagating an edge and a
structure in the first instance.
One of the major problems in exemplar-based method is
a process of searching the patch with the maximum
similarity to a selected patch using mean squared error
metric. As a result, the algorithm will produce visually poor
result. Thus, the criterion to search the best match, such as
the Euclidean metric for pixel brightness, may lead for some
images to uncorrected reconstruction since a method ignores
the textural characteristics of the patches.
The purpose of this work is to modify an exemplar-
based method in order to overcome above mentioned
drawbacks of EBM. We propose to use multiple criteria for
a patch similarity search since often in practice existed
exemplar-based methods produce unsatisfactory results. The
criteria for searching the best matching uses several terms,
including Euclidean metric for pixel brightness and chi-
squared histogram matching distance for local binary
patterns. A combined use of textural geometric
characteristics together with color information allows to get
more informative description of the patches.
We summarize the algorithm in the following scheme
(figure 2). Fig.2. The proposed inpainting algorithm.
Below we will describe all building blocks of the
proposed algorithm. 3.2. Defining a pixel with a maximum priority

3.1. Calculating the priority Define a pixel with a maximum priority on the border of the
missing region.
A priority value is computed for each value of the pixel
boundary consisting of the confidence term and the data 3.3. Defining a restored patch
term. Calculating the priority using the expression allows
giving larger weights to pixels that are on the differences of Define the square patch Ψ p with a center in the pixel with a
brightness (the boundaries), thus restoring them first. The maximum priority.
confidence term sets lower weights to the restored pixels
with increasing distance from the true pixels of the region. 3.4. Calculating Euclidean metrics for true pixels and
A priority value P(δS ) is computed for each value of the restored patch

659
On the true image S we find a patch ψ q , for which the
Euclidean metric is minimal:
DE (Ψ p , Ψq ) = ∑ (Ψ p − Ψq ) 2 
→ min .
3.5. Calculating the LBP
We have utilized the local binary patterns (LBP) as a texture
descriptor for patches in image. The basic LBP operator is
calculated by a thresholding of each pixel around the central
pixel in a local 3x3 pattern [8].
3.6. Forming a histogram
Among all possible binary patterns formed by LBP operator,
only 9 uniform patterns are used for analysis. Therefore, the
histogram of the uniform patterns in a local region forms a
powerful descriptor used to analyze the local regions in
image. The histogram is formed for every local area Ψ p , Ψq
and used as a feature descriptor.
3.7. Calculating the Chi-squared histogram matching
To determine the correspondence between the histograms
have been proposed the Chi-squared histogram matching
distance in [9]:
1
K h (
Ψ p ( m) − hΨq ( m)
2
) A main feature of test image «Flowers» is the fact that
2
χ (hΨ p , hΨq ) = ⋅ ∑
2 m =1 hΨ p (m) + hΨq (m)
, the regions with missing pixels located at the intersection of
where hΨ p and hΨq are the histogram of LBP for Ψ p and
Ψq patches, respectively.
3.8. Searching similar patch using multiple criteria
For the multiple criteria, we have used Euclidean metric for
pixel brightness and Chi-squared histogram matching
distance for local binary patterns in the form:
F = (1 − β ) ⋅ DE ( Ψ p , Ψq ) + β ⋅ χ 2 ( hΨ p , hΨq ) 
→ min ,
where β is a weight coefficient.
A proper selection of the coefficient β in multiple
criteria proposed in [9]. This weighting factor can vary from
0 to 1. If β = 0 then as a measure of proximity for
searching a similar patch only Euclidean metric is used, and
if β = 1 then we use chi-squared histogram matching
distance for local binary patterns. Analysis of the
experimental results demonstrate that the modified
exemplar-based method may reduce the error of image
reconstruction in average by 18% when the weight
coefficient is set to β = 0,25 .
660
3.9. Patch restoration using minimum error boundary cut
To reduce the visibility boundaries on the reconstructed
image between similar patches in EBM we use the
algorithm for texture synthesis proposed by Efros and
freeman instead of simple copy-and-paste procedure. It
allows optimizing an overlap region between patches using
minimum error boundary cut (figure 3).
The confidence data for the restored pixels is assigned to
the current value C ( p ) . After this a procedure of calculation
of priority and searching for similar patches with subsequent
replacement repeats.
ψ p ψ q ψ p
ψ q
Fig. 3. Minimum error boundary cut.
4. RESULTS
In figures 4-6 examples of an image reconstruction (a -
original image, b - the image with a mask, c - the image
reconstructed by the Telea method (TM), d - the image
reconstructed by the EBM, e - the image reconstructed by
the proposed method) are given.
the curve boundaries that need to be extrapolated, are
connected to the opposite border. The proposed method
allows the correct reconstruction of pixel values. The TM
blurs same regions and the EBM is fail in restoration in the
absence of similar blocks.
A feature of the test image «Lighthouse» is that the area
with missing pixels is located at the intersection of the
borders separating the texture and non texture. The results
show that the proposed method can correctly restore the
structure and texture regions.
Others examples presented in this paper demonstrate the
effectiveness of the algorithm in restoration of the test
images having different geometrical characteristics.
The proposed method allows to synthesize the texture
with a simultaneous reconstruction of an image structure.
It's worth noting that the method does not smear the texture
and the structure during the restoration of large areas of
missing pixels.
Table 1 shows numerical comparison of three methods,
TM, EBM and the proposed one, in terms of Root Mean
Square Error (RMSE). Note, that for EBM parameters are set
to provide best numbers for the considered test images. It is
worth noting that the error values confirm the visual analysis.
the proposed method provides smaller reconstruction errors,
on average 30-80% less than the processing of EBM and
TM.
 5. CONCLUSION

This paper presented an image inpainting algorithm

based on a modified exemplar-based technique. It is
proposed to use a multiple criteria to search a similar patch
which uses two terms: the Euclidean metric for pixel
(a) (b) brightness and Chi-squared histogram matching distance for
local binary patterns. A use of textural geometric
characteristics along with the color information resulted in
better description of the patches. Several examples
presented in this paper demonstrated the effectiveness of the
algorithm in the image reconstruction. The proposed
technique is able to produce visually preferable results in
(c) (d) (e) reconstruction of missing small and large objects on test
Fig. 4. Reconstruction of the image «Flowers». images than the exemplar-based method and Telea method.

REFERENCES

[1] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and

J. Verdera, “Filling-in by join interpolation of vector
fields and grey levels,” IEEE Trans. Image Process.,
vol. 10, no. 8, pp. 1200–1211, 2001.
[2] A. Telea, “An image inpainting technique based on the
fast marching method," Journal of Graphics Tools, vol.
9, no. 1, ACM Press, pp. 25-36, 2004.
(a) (b) [3] J.L. Starck, M. Elad, and D.L. Donoho, "Image
decomposition via the combination of sparse
representations and a variational approach," the IEEE
Trans. On Image Processing, vol. 14, no. 10, pp. 1570-
1582, 2005.
[4] Li Xin, “Image Recovery via Hybrid Sparse
Representations: a Deterministic Annealing Approach,”
Selected Topics in Signal Processing, IEEE Journal of,
vol. 5, Issue 5, pp. 953 – 962, 2011.
[5] Chen Yan, Luan Qing, Houqiang Li, O. Au, “Sketch-
Guided Texture-Based Image Inpainting,” Image
(c) (d) (e)
Fig. 5. Reconstruction of the image «Astro». Processing, 2006 IEEE International Conference on,
pp. 1997 – 2000, 2006.
[6] A. Criminisi, P. Perez, and K. Toyama, “Region filling
and object removal by exemplar-based image
inpainting,” IEEE Transactions on Image Processing,
vol. 13, pp. 1200–1212, 2004.
[7] V.V. Voronin, V.I. Marchuk, and K.O. Egiazarian,
“Images reconstruction using modified exemplar based
(a) (b) (c) (d) (e) method,” in SPIE Electronic Imaging (2011)], 7870,
Fig. 6. Reconstruction of the image «Lighthouse». 2011.
[8] T. Ojala, M. Pietikäinen, T. Mäenpää, “Multiresolution
Table 1. The error of processing for different images
gray scale and rotation invariant texture analysis with
RMSE
Local Binary Patterns,” IEEE Transactions on Pattern
Images TM EBM Proposed method
Analysis and Machine Intelligence, vol. 24, no. 7, pp.
«Flowers» 12.544 10.061 6.956
971–987, 2002.
«Astro» 13,352 11,62 8.477
[9] V.V. Voronin, V.I. Marchuk, N.V.Gapon, R.A.
«Lighthouse» 11,913 9,913 7.179
Sizyakin, A.I. Sherstobitov and K.O. Egiazarian,
“Exemplar-based inpainting using local binary
patterns,” in SPIE Electronic Imaging (2014)], 9019,
2014.

661