You are on page 1of 6

IMAGE COMPRESSION USING INPAINTING Min-Wook Jeong, Young Min Kim, and Jungsuk Kwac

ABSTRACT This paper examines a way to enhance the compression ratio of still images by deleting parts of an image prior to transmission and recover them at receivers side. The image is initially divided into 8 by 8 macro blocks. Blocks to be erased will be classied as either structure or texture depending on their own characteristics and relation to adjacent blocks. Each erased block will be recovered by using an appropriate reconstruction method. Quantitative analysis of the improvement in compression ratio and the quality of the restored image will be conducted. The overall rate distortion performance of this method will be compared with that of JPEG and JPEG2000. Index Terms Inpainting, structure, texture, block classication, edge detection, compression ratio, peak signal-tonoise ratio (PSNR) 1. INTRODUCTION Image inpainting is a technique of lling in the missing region of an image. The main usage of inpainting is in restoring damaged part of the picture. [1] This implies that without transmitter sending entire picture the receiver can still recover missing part of the image to come up with the whole image in the end. We can think of the transmitter intentionally not sending certain block of image and the receiver lling in the missing region upon reception of the incomplete picture. By not sending signicant portion of the picture that can later be restored from remaining part, we can signicantly reduce the amount of bits needed to transmit the image. For this method to be successful, it is important to choose and erase the block that can be easily restored. There are two types of regions that can be relatively easily reconstructed, structure and texture. It is also important to properly classify these blocks into either of these two. As this method presupposes that missing blocks can be accurately recovered by appropriate inpainting algorithm, careful attention should be paid to the inpainting algorithm for each type of blocks. This paper is organized as follows. First, previous work on image inpainting will be reviewed. It will be followed by the description of our inpainting algorithm. O Our algorithm consists of four parts, 1) classication of blocks, 2) selection of blocks to be erased, 3) texture inpainting, 4) structure inpainting. Each part will be explained in detail. O Experimen-

tal result of our algorithm combined with JPEG will be compared with compression using JPEG alone. This paper will end with general conclusion attainable from our experiment. 2. PREVIOUS RELATED WORK In essence, image inpainting is quite different from denoising or image enhancement. Unlike common image enhancement applications, in which pixels contain both information about the real data and the noise, in image inpainting, the only information available for reconstruction is the average value of the erased block. Therefore, it is necessary to develop new techniques to address these problems. With regard to image inpainting, there used to be mainly three types of methods. The rst deals with the restoration of lms. The second is related to structure and texture lling in missing image blocks, and the last is related to disocclusion. We will focus on the second method, which is most related to our project. In [1], the method of restoring damaged picture through inpainting is introduced. The algorithm is based on solving partial differential equations similar to that of diffusion using pixel value gradient of neighboring blocks. In [2], restoration schemes for two different types of missing region, structures and textures, are described. Methods of classifying blocks into structure or texture is also suggested. Based on these previous works, we focus to improve our algorithm in four different aspects described below. 1) Further classifying structure as smooth area and weak edge. The inpaint method used for each will slightly vary. 2) Use of Canny edge detection to better recognize sharp edges. 3) Exploiting statistical similarity in classifying block as texture, in addition to using coarseness comparison. 4) Extra methods of lling in textures. 3. PROPOSED ALGORITHM The algorithm used for image compression using inpainting is composed of four major parts: 1) Classify each block into texture or structure according to the algorithm to be used to reconstructed. 2) Choose blocks to be removed. 3) Fill in removed blocks classied as texture. 4) Fill in removed blocks classied as structure. The four algorithms will be described in the order.

3.1. Block Classication The whole algorithm runs in units of 8x8 blocks as JPEG divides images into 8x8 blocks for compression. The blocks are classied in two steps: We rst check whether each block is recoverable using either texture or structure inpainting. Then, the blocks to be removed and appropriate algorithm for their later reconstruction are selected based on the analysis of the rst step. For decision, the relationship with surrounding eight blocks is critical because the neighboring blocks provide the information needed for reconstruction of a missing block. Texture inpainting successfully reconstructs a missing block when surrounding blocks have similar statistical characteristics. Such similarity is briey checked with mean and variance of each of neighboring blocks. If the mean and variance are similar to surrounding blocks, then the block in the middle is recoverable with texture inpainting. In contrast to texture inpainting, which has a limited application for lling in missing region, structure inpainting successfully reconstructs image in general. We can apply structure inpainting if a block falls into one of the two cases: 1) when a block does not contain strong edges, and 2) when it is not composed of ne repetitive patterns. Sharp (strong) edges are critical when human recognize the shape of an object. Structure inpainting blurs the sharp edges because the algorithm is based on diffusion equations. Also, ne details are not recoverable because the diffusion equation continuously connects the missing region with surrounding region. The case of 1) can be found using Canny edge detector [3] and 2) can be found by algorithm proposed in [2].

Figure 1. An example of mask on pepper (black : texture, gray : structure with edge, white : at structure) Figure 1 shows an example of mask found by proposed algorithm. Note that large portion of at area is removed while alternative blocks are missing for other cases. Blocks with ne variation or sharp edges are conserved. 3.3. Texture Inpainting We classify blocks whose statistical properties are similar to those of surrounding blocks into texture. So, when we want to ll the texture block, we exploit the statistical similarity with other blocks. (Out texture synthesis algorithm is not much different from the previous work.) Briey speaking, texture inpainting is to nd best match from referable surrounding blocks in statistical aspect. In detail, texture synthesis process is like below. At rst, when we call texture synthesis function, we give referable neighborhood block information. Secondly, we set the template which is 3x3 or 4x4 and next to the missed pixel we want to ll. We use this template as the best match criteria. Using MMSE algorithm, we nd the best match among referable neighborhood blocks, and then, copy the pixel next to the missed pixel. Filling order is from up to down and from left to right. Finally, to make same DC value as before, we normalized the pixel values of the lled texture block. This texture synthesis method is similar to that of the previous works. So, there is another method, we used. In big picture(512x512), 8x8 block is not a big portion. And, in smooth area, we dont use texture synthesis because of PSNR quality, even though texture synthesis is faster than structure inpainting. In most cases, we use texture synthesis in very coarse area or pattern area. From those reasons, we copy the whole block(8x8) to the missed texture block, after nding

3.2. Removing Blocks Our goal is to remove as much blocks as possible and still recover the image with little perceptual difference. Based on the classication of part A, we designed an algorithm to automatically nd suitable masks using following general rules: 1) For blocks recoverable using texture inpainting, remove alternative blocks because we need surrounding blocks for texture inpainting. 2) Remove as much blocks in smooth area that can be restored using structure inpainting as possible. Such blocks are not noticeable even we simply ll in DC value. 3) Remove alternative blocks along an weak edge since we need neighboring blocks to connect the edge. 4) Fill in the removed blocks with mean pixel value before removal.

the block which has the closest mean and variance to missed texture block. In this case, we can get a good result in visual aspect, despite of almost same result in PSNR. 3.4. Structure Inpainting Structure is the region that can be clearly divided into two or more sectors through clear edges. Each sector is relatively devoid of minute details and the missing block within the sector is easily predicted from surrounding blocks. To state simply, structure inpainting is the process of gradually propagating the information contained in the surrounding blocks into the missing block. Basically, this process is actually very similar to that of diffusion. When the heat source (surrounding block) is placed around closed area (missing block), heat (information) gradually ows into the area. Now the problem reduces to 1. nding the right information to propagate and 2. determining proper propagation direction. Let I(x, y) denote the pixel value of the image at (x, y). According to [1], we will use for information L(x, y), the Laplacian of the image I(x, y). As x Z and y Z, to calculate Laplacian we have to use discrete implementation of Laplacian [4] L(x, y) = I(x + 1, y) + I(x, y + 1) + I(x 1, y + I(x, y 1) 4I(x, y)

steady state value I(x, y, ) depends on the characteristics of known adjacent blocks. If the missing block happens to be a part of edge, then we denitely need a lot of iterations to have the steady state value, whereas for at regions we need relatively small number of iterations.
R (Assistant Information) R (Compressed Assistant Information) D (Compress ed Image)

I (Original Image)

Image Analysis & Block Removal

Lossless Encoder

D (DC Filled Image)

JPEG Encoder


Lossless Decoder

R (Assistant Information)

JPEG Decoder

D (DC Filled Image)

Filling in Removed Blocks

I (Original Image)


As for the direction of propagation, propagating the information in the direction inward normal to the boundary of the block is not a good scheme as it cannot preserve the direction of edges. One reasonable alternative from [1] is to propagate information in the direction of isophote or normal to that of I(x, y) at (x, y). By denoting vector of propagation direction N (x, y), we have from [4] N (x, y) = = dI(x, y) d(I(x, y) , dy dx I(x, y + 1) I(x, y 1) [ , 2 I(x + 1, y) I(x 1, y) ] 2

Figure 2. An algorithm for image compression using inpainting 4. EXPERIMENTAL RESULT The steps of compression are depicted in Figure 2 and the resulting image for each step is shown in Figure 3. The original image I is analyzed and the blocks to be removed are determined such as shown in previous section (Figure 3(a)). The assistant information R should be sent with masked image to decoder. R should contain the locations of blocks removed and the algorithm to be used to ll in the missing region. For missing region, ll in DC values to minimize the size of JPEG-encoded image. R is compressed using lossless encoder while DC-lled image D (Figure 3(b)) is encoded by JPEG. On the decoder side, R and D are decoded and R is used to ll in the removed blocks of D. The bit rate is calculated by (size of D) + (size of R (entropy encoded R)) / (size of the image) [BPP]. The BPPPSNR is compared with JPEG of the similar BPP in Figure 4(BPP-PSNR.jpg). We see about 0.1 0.3dB improvement in PSNR for same BPP. The image with similar quality but higher bit-rate using JPEG alone is provided in Figure 3(d) for comparison with the reconstructed image using image inpainting (Figure 3(c)). In this example, the BPP is lowered by

Therefore, the time rate of change of the pixel value I(x, y) is described as dI(x, y) = C L(x, y) N (x, y) dt and in discrete form as I(x, y, t + 1) = I(x, y, t) + C L(x, y) N (x, y), where t is the time or iteration index. The value of C can be freely modied as long as the equation does not diverge. As this equation gradually lls in the missing region, the number of iterations needed to make I(x, y, t) converge to

22.8%. Other examples are given in Figure 5 and Figure 6. Again, the same PSNR can be achieved using our algorithm with lower BPP (Table 1). 5. CONCLUSIONS AND FUTURE WORK In this paper we tried some new techniques for classifying structure and texture block. From that, it was natural to try new techniques to recover the missing blocks. Greatly, we could get better result(0.1 0.2dB better PSNR at xed bpp) than JPEG in gray images. Furthermore, we tested our algorithms for color image. Even though it is difcult to decide which, the masked image and reconstructed image, is better in visual aspect, in PSNR aspect, the reconstructed image was better than masked image. By adding our own ideas to previous works algorithms, we improved the performance of image inpainting. This is our bottom line. And there are many things to look worthy to try. It would be good to try to new method to masking method for bigger compression ratio. Besides, it would be very important to make the processing time cost in structure inpainting faster.

6. REFERENCES [1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, SIGGRAPH, pp. 10331038, January 1999. [2] S.D. Rane, G. Sapiro, and M. Bertalmio, Structure and texture ling-in of missing image blocks in wireless transmission and compression applications, IEEE Transactions on Image Processing, pp. 296303, March 2003. [3] J Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. Intell., vol. 8, no. 6, pp. 679698, 1986. [4] E. Kreyszig, Advanced Engineering Mathematics, John Wiley & Sons, Inc., 1999.



Figure 5. Example of aireld (a) masked image (b) reconstructed image



Figure 3. Stages of encoding and decoding with pepper. (a) masked image (b) DC-value lled image (0.78bpp) (c) Reconstructed image (PSNR=38.61dB) (d) Image using JPEG only (1.01bpp, PSNR=38.60dB)

Figure 6. Example of boats (a) masked image (b) reconstructed image

Image peppers peppers aireld boats bridge harbour lena(color) Size BPP 256*256 512*512 512*512 512*512 512*512 512*512 128*128 1.23 0.81 1.37 0.83 3.02 1.99 2.38 JPEG CR 6.50 9.88 5.84 9.64 2.65 4.02 10.08 PSNR 39.88 38.59 34.40 39.90 39.98 39.88 33.86 JPEG with Inpainting BPP 0.84 0.78 1.30 0.84 0.84 0.84 2.24 CR 9.52 10.26 6.15 9.52 9.52 9.52 10.71 PSNR 39.97 38.60 34.43 39.97 39.97 39.96 33.93

Table 1. Improvement in BPP and PSNR

Figure 4. BPP-PSNR curve of proposed algorithm and JPEG on pepper

Min-Wook Jeong : 1. Implementing MATLAB source code for structure inpainting. 2. Writing abstract, introduction and structure inpainting part of project report 3. Putting together portions of project report written by team members 4. Preparing slide for presentation and conducting presentation

Young Min Kim 1. Implementing MATLAB source code for block classification and selecting blocks to remove. 2. Putting together portions of MATLAB source codes written by other team members 3. Writing block classification, selection and experimental result part of project report 4. Running simulations

Jungsuk Kwac 1. Implementing MATLAB source code for texture inpainting part 2. Improvement of selection algorithm 3. Writing previous work, texture inpainting, conclusion and future work of project report 4. Running simulations ( Color Image )