VIPromCom-2002,4th EURASIP - IEEE Region 8 International Symposium on

Video / Image Processing and Multimedia Communications, 16-19 June 2002, Zadar, Croatia

FAST ADAPTIVE UPSCALING OF LOW STRUCTURED

IMAGES USING A HIERARCHICAL FILLING STRATEGY
Serap Askar, Peter Kauff, Nicole Brandenburg, Oliver Schreer

Heinrich-Hertz-Institut fur Nachrichtentechnik Berlin GmbH,

Image Processing Department,
Einsteinufer 37, D-10587 Berlin, Germany
Email: { askar, kauff, brandenburg, schreer }@hhi.de

Abstract: Image upscaling approaches are required especially when algorithms are applied
to subsampled images in order to speed up processing time. Conventional techniques for
upscaling images yield often blurred or blocky results, but more complicated methods are
time consuming. We present an adaptive upscaling approach for low structured images,
which provides real time capability for images in full TV resolution and produces clear and
non-blocky edges due to a hierarchicalfilling strategy.

Key words: adaptive interpolation, subsampling, real time video processing, binary mask
interpolation, dense disparity estimation

1. INTRODUCTION

Various applications need interpolation approaches for resizing images. When resizing
images the main challenge is how to reconstruct adequate values for missing pixels in the
large image using information from pixels in the small image. Typically used approaches
here are nearest neighbor method, bilinear interpolation and bicubic interpolation [11-[7].
One main drawback of all these methods, however, is that they produce either blocky edges
(nearest neighbor) or blurred images (bilinear and bicubic interpolation). Furthermore, if
there is an additional real time constraint, these techniques are too complex as well as being
too slow. Therefore we present a novel approach, especially for low structured images or
binary images, which is based on a fast interpolation technique using a new hierarchical
filling strategy for preserving clear edges while providing real time capability in full TV
resolution video sequences.

2. HIERARCHICAL FILLING STRATEGY

The main problem with conventional interpolation techniques is that they apply the same
interpolation procedure to the whole image. Hence, costly procedures cannot be realized in
real time whereas simple techniques do not reach sufficient quality. Therefore we propose an
adaptive approach, which distinguishes between image regions of different criticality. In our
approach a computationally intensive and edge preserving technique is only performed in
critical image areas, whereas simple and more straight-forward methods are used in the
remaining regions.
The adaptive interpolation approach we are presenting in this paper works blockwise. For a
NxN up-sampling factor NxN blocks are interpolated by processing (N+l)x(N+l)

289

Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.
 VIPromCom-2002,4th EURASIP - IEEE Region 8 International Symposium on
Video I Image Processing and Multimedia Communications, 16-19 June 2002, Zadar, Croatia

overlapping windows. In each block the pixel values at the corners are known from the small
image (see Fig. 1 and Fig. 3). For determining the missing pixel values in a current block
only pixels inside this block are considered. The decision on a suitable interpolation
technique is made by analyzing the correlation of the corner pixels in a block. If all four
values are similar, or even equal, a standard interpolation method technique is used.
Otherwise a hierarchical filling technique is applied. For this latter technique it is assumed
that the image is low structured, i.e. there are at the utmost two regions in a block,
representing the two sides of an edge. Furthermore it must be taken into account, that the
block size, representing the up-sampling factor is a power of two.
In the next chapters two examples are presented, describing the hierarchical interpolation
approach for a binary image case and a gray value image case, respectively.

2.1. Filling Binary Images

In Fig. 1the up-sampling of a binary image with an up-sampling factor 4 is represented.

homogenous
transition transition

i
black

white

0
pels to
be filled

homogenous regions

Fig. 1: Interpolation of a binary image

Homogeneous and inhomogenous areas are marked pointing out two different filling
methods. Homogeneous regions recognized by detecting identical corner pixels in a block,
are filled by a simple copy of the reference pixel value into the missing intermediate pixels.
In inhomogenous regions at least one of the 4 corner pixels differs from the others (Fig. 2). In
this case filling works as follows. The first row and the first column in a window with
varying corner pixels are known from previously processed overlapping blocks.

step 1 step 2 step 3 step 4

Fig. 2: Hierarchical filling of a binary block

As depicted in Fig. 2, in the first step of the hierarchical filling the pixel at central block
position is filled (marked gray in step 1 of Fig. 2). Here only the 4 corner pixels are taken as
decision candidates for a majority decision criterion. If the result is inconclusive, two
solutions are possible. The first one is to pick out one label in advance before the whole

290

Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.
 VIPromCom-2002,4th EURASIP - IEEE Region 8 International Symposium on
Video I Image Processing and Multimedia Communications, 16-19 June 2002, Zadar, Croatia

processing and to decide in favor of this label in such situations. The other is to take
additional information into account, if possible, of course depending on the application.
Because of the fact that this middle pixel carries a quite significant influence on the
succeeding processing, it is recommended to take further information into consideration (e.g.
frame difference at this mid-point position in the case of change detection or segmentation
issues). In the second step the same procedure is repeated for the four resulting sub-blocks,
resulting in the four new mid-position pixels. For each of these pixels the decision is made on
basis of the two direct neighbors in diagonal direction. In the third step only horizontal and
vertical neighbors are considered. In the last step for each pixel all four direct neighbors (two
in horizontal and two in vertical direction) are considered. Note that, as a result of this
strongly hierarchical procedure, the interpolation is fast and delivers a conclusive and smooth
edge.

2.2. Filling Gray Value Images

Fig. 3 shows the up-sampling of a gray value image with an up-sampling factor of 4.

block with
strongly varying
corner values block with similar
corner values

edge

homogeneous region

Fig. 3: Interpolation of a gray value image

In Fig. 3 (left) a disparity map is shown as an example for a low-structured image. On the
right one window with quite different corner pixels and one window with similar corner
pixels are chosen. A threshold for the difference between the lowest and highest corner pixel
must be defined in order to decide whether to treat the block as a homogeneous or an
inhomogenous block. Homogeneous blocks can be filled e.g. by bilinear interpolation or
nearest neighbor approach assuming a uniform area. Inhomogenous blocks are filled
hierarchically as shown in Fig. 4.

steD 1 steu 2 steD 3 step 4

Fig. 4: Hierarchical filling of a gray value block

29 1

Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.
 VIPromCom-2002,4th EURASIP - IEEE Region 8 International Symposium on
Video I Image Processina and Multimedia Communications, 16-19 June 2002, Zadar, Croatia

The filling strategy is quite similar to the binary case. In the first step the middle pixel is
filled taking into account the information from the four corner pixels. The four regarded
pixels can have four different values. In order to keep edges clear and unblurred a method
derived from median filtering is applied. The determination of a missing pixel value in an
inhomogenous block is done in gray value images by sorting all decision candidates
numerically and then taking the pixel value in the middle of this list as result. If there is an
even number of candidates, one of the values in the middle of the list must be favored in
advance. In the second step the marked pixels are filled using information from horizontal
and vertical neighbors. In step 3 four direct diagonal neighbors are considered to fill pixels of
that hierarchy. And in the last step the last missing pixels are filled from direct horizontal and
vertical neighbors. In all steps the median filtering based idea is applied, to determine
missing pixels. The advantage of this approach is that no blurring is generated but edges can
be stressed by copying pixels.
The examples above are given for an up-sampling factor of 4. But they can also be applied to
higher up-sampling factors, breaking the respective blocks hierarchically down to 4x4 blocks.

2.3. Experimental Results

Fig. 5 shows experimental results from two different applications. The binary images show
masks as a result of a segmentation process on CCIRBOl format, which has been applied to
the subsampled image to speed up the processing. The middle image shows the resizing
result using nearest neighbor, whereas on the right the non-blocky result of hierarchical
filling is presented.

Fig. 5: Top: segment masks, Bottom: depth maps;

subsampled images (left), simple interpolation (middle), hierarchical approach (right)

292

Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.
 VIPromCom-2002,4th EURASIP - IEEE Region 8 International Symposium on
Video I Image Processing and Multimedia Communications, 16-19 June 2002, Zadar, Croatia

The gray value images show up-sampling results of a disparity map (left), which has been
estimated on an 8x8 grid. While the middle image created by bilinear interpolation shows
blurring effects, the right result preserves edges clear and interpolates homogeneous areas.
In [8]for example this hierarchical interpolation approach is applied to obtain dense disparity
maps from sparse disparity maps in real time.

3. CONCLUSION

We presented a novel approach for fast upscaling of low structured images, which produces
smooth but unblurred and non-blocky edges. This goal is achieved by the adaptive character
of the approach applying a computationally intensive method only in critical areas and a
simple method in the other areas. In the critical regions filling in a hierarchical order, and for
gray value images using a median filtering base, preserves edges clear. In the other regions a
less costly method simply interpolates the area.

REFERENCES

[ 11http://www.fmde.com/multi-tech-86 lO-image-interpolation_methods.htm
[2] J.-R. Ohm, “Digitale Bildcodierung“, Springer-Verlag, Berlin, 1995
[3] J.-R. Ohm, “Bildverarbeitung 11“, Skript, TU-Berlin, Institut fiir Nachrichtentechnik und
Theoretische Elektrotechnik, 1999
[4] http://www.pcigeomatics.com/cgi-bin/pcihlp/REGlBACKGROUND
[5]http://www.cee.hw.ac.uk/hipr/html/scale.html
[6] http://www.dpreview.com/learn/Glossary/Digital_Imaging/lnterpolation-01.htm
[7] http://www.mathworks.com/access/helpdesk/help/toolbox/images/geom2.shtml
[8] 0.Schreer, N. Brandenburg, S. Askar, P. Kauff, “Hybrid Recursive Matching and
Segmentation-Based Postprocessing in Real-Time hmersive Video Conferencing”, Proc. of
VMV 2001, Vision, Modeling and Visualization 2001, Stuttgart, Germany, Nov. 2001

293

Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.
Authorized licensed use limited to: TU Hamburg-Harburg. Downloaded on September 09,2020 at 23:05:59 UTC from IEEE Xplore. Restrictions apply.