You are on page 1of 20

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 1 Introduction
Vision is the most advanced of our senses and images play an important role in human perception. To obtain accurate information in many imaging applications, a high resolution (HR) image is required. The demand for high-resolution images is immense in applications such as industrial inspection, remote sensing, military surveillance and medical diagnostics. Such images offer additional details that may be critical in accurately analyzing the image. For example, high resolution image is helpful to make a correct diagnosis in medical applications, easier to distinguish an object from similar ones in satellite images, improving the performance of pattern recognition in computer vision.

1.1 Image resolution:


An image is defined as a two-dimensional function, f(x,y), where x and y are spatial coordinates, and the amplitude of f at any pair of coordinates (x,y) is called the intensity of the image at that point [1]. When x, y, and the amplitude values of f are all finite discrete quantities, image is called a digital image. It is composed of a finite number of elements, each of which has a particular location and value. These elements are referred to as picture elements, image elements , pels, and pixels. Image resolution is defined as the smallest measurable detail in a visual presentation. Resolution is defined in three different ways: spatial resolution, brightness resolution and temporal resolution. Spatial resolution refers to the spacing of pixels in an image and is measured in pixels per inch. Larger the number of pixels, higher is the resolution and smaller will be the size of individual pixels. Fig. 1.1(a) shows an MRI image of 256 X 256 pixels. When represented with 64 X 64 pixels as shown in Fig. 1.1(b), results in pixellization where individual pixels can be seen. Brightness resolution refers to the number of brightness levels that can be recorded at any given pixel. Temporal resolution refers to the number of frames captured per second and is also commonly known as the frame rate. It is related to the amount of perceptible motion between the frames. In this project resolution refers to spatial resolution.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 1

PROJECT TITLE

2011-12

__________________________________________________________________

(a)

(b)

Fig. 1.1 Image with spatial resolution of (a) 256 X 256 pixels, (b) 64 X 64 pixels

As an alternative to expensive high-resolution cameras, signal processing approaches are typically used to enhance the resolution. A single high resolution image from low resolution frames is possible only if they are sub-sampled (aliased) and have sub-pixel shifts. Several super-resolution techniques have been proposed in the literature. A brief discussion of these techniques is given here.

1.2 Literature survey:


Existing super-resolution algorithms can be broadly divided into three categories: motion-based, motion-free and learning based. In motion based super-resolution, a high resolution frame can be obtained from sub-pixel shifted low resolution frames [5], [6]. Motion free super-resolution is reconstruction of a high resolution image with differently blurred images without relative motion [7]. In learning based techniques, high frequency features are learnt from a set of high resolution images and then used to super-resolve a low resolution image [8], [9].

Motion based multi-frame super-resolution techniques are broadly classified into frequency domain methods and spatial domain methods [10]. Frequency domain super-resolution restoration methods typically rely on familiar Fourier transform properties like shifting and sampling theorems. It makes use of aliasing that exists in each low resolution image to construct a high resolution image. These methods offer advantages like simplicity and less computational _____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 2

PROJECT TITLE

2011-12

__________________________________________________________________
complexity. A brief explanation of frequency domain techniques follow (a) Restoration via alias removal: It is a frequency-domain formulation based on the shift and aliasing properties of the continuous and discrete Fourier transforms for the restoration of a single bandlimited image from the set of under-sampled and therefore aliased observed images [11]. (b) Recursive least square methods:This is an extension of frequency-domain restoration proposed by Kim et. al. [13] called weighted least squares formulation. (c) Recursive total least squares methods: Bose et. al. [15], [16] proposed the recursive total least squares method for SR reconstruction to reduce effects of registration errors. (d) Multichannel sampling theorem methods: This is a super-resolution method based on the generalized sampling theorem and multichannel sampling theorem [17]. The other major category of multi-frame super-resolution restoration comprises methods which are based on a spatial-domain formulation and solution of the restoration problem. The major advantage of these spatial-domain methods is that they provide generality regarding observation models and powerful methods for the inclusion of a-priori constraints. Different spatial domain approaches are as follows: (i) Interpolation of non-uniformly spaced samples: This is a simple approach to construct super-resolved image from an image sequence. (ii) Algebraic filtered back-projection method: In this method, the restoration problem is formulated as discrete, linear inverse problem which is solved in a single step. A back projection operator, which is often the adjoint operator of the linear operator approximating the observation model, is used in the solution, thus lending the name "back projection". (iii) Iterative back-projection method: Irani and Peleg [25] formulated the iterative backprojection (IBP) super-resolution reconstruction approach that is similar to the back projection _____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 3

PROJECT TITLE

2011-12

__________________________________________________________________
used in tomography. (iv) Stochastic methods: Stochastic SR image reconstruction, typically a Bayesian approach [19], provides a flexible and convenient way to model a priori knowledge concerning the solution. (v) Set theoretic methods: These methods solve the restoration problem by defining sets of constraints which must be satisfied by candidate solutions.

1.3 Scope of the project:


Theoretical and practical limitations usually constrain the achievable resolution of any imaging device. Specifically, the imaging chips and optical components necessary to capture very high resolution images become prohibitively expensive, costing in the millions of dollars for scientific applications. Hence, there is a growing interest in the multi-frame image reconstruction algorithms that compensate for the shortcomings of the imaging systems. Such methods can achieve high quality images using less expensive imaging chips and optical components by capturing multiple images and fusing them. The application of such algorithms help in any situation where high quality optical imaging systems cannot be incorporated or are too expensive to utilize. Super-resolution is the algorithmic way to overcome the limitations of the optical imaging systems through the use of image processing algorithms, which are relatively inexpensive to implement.

Applications:
Applications of super-resolution restoration from image sequences appear to be growing rapidly as the theory gains exposure. Super-Resolution restoration techniques can be applied to problems in: Satellite imaging Astronomical imaging Video enhancement and restoration

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 4

PROJECT TITLE

2011-12

__________________________________________________________________
Video standards conversion Confocal Microscopy Aperture displacement cameras Medical imaging Robotics and many more fields where it is necessary to increase the overall information processing power.

1.4 Organization of the report:


Chapter 2 provides a brief review of super-resolution and different techniques. In Chapter 3, iterative back projection algorithm is studied. Dealiasing capability of super-resolution algorithm is verified in Chapter 4 on one dimensional signals. In Chapter 5 results obtained using IBP algorithm under different conditions of blur and noise on images is given. With application specific DSP processors yielding better performance in real time applications, Chapter 6 gives implementation of the algorithm using DSP processor TMS320C5416. Algorithm and flowchart for the software are given in Chapter 7. Report concludes with Chapter 8 wherein suggestions for future work are also indicated.

Chapter 2

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 5

PROJECT TITLE

2011-12

__________________________________________________________________

Super-resolution
Super-resolution refers to obtaining high resolution image from a set of low resolution observations. The basic idea behind super-resolution is the fusion of a sequence of low resolution blurred noisy images to produce a higher resolution image. The major advantage of this signal processing technique is that it costs less and existing low resolution imaging systems can still be utilized. In this chapter, we study the basic concepts of super-resolution. The concept of image zooming is explained to convey that re-sizing cannot improve resolution. Hadamard conditions are used to show why super-resolution is ill-posed inverse problem.

2.1 Image zooming:


Re-sizing of the image does not translate into an increase in resolution. In fact re-sizing should be accompanied by approximations for frequencies higher than those representable at the original size and a higher signal to noise ratio. This process of re-sizing for increasing resolution is called upsampling or image zooming. The traditional method of upsampling is to use interpolating functions wherein original data is fit with a continuous function and then resampled at a finer sampling grid. In implementing resampling, interpolation and sampling are often combined so that the signal is interpolated at only those points which will be sampled.

(a)

(b)

Fig. 2.1 Example of image zooming (a) 128 X 128 image, (b) 256 X 256 image obtained by interpolation

The quality of the interpolated image generated by any of the single input interpolation algorithms is inherently limited by the amount of data available in the image. Image zooming cannot produce the high frequency components lost during the low resolution sampling process unless a suitable model for zooming can be established. Fig. 2.1(a) shows a 128 X 128 image. Bilinear interpolated _____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 6

PROJECT TITLE

2011-12

__________________________________________________________________
256 X 256 image is shown in Fig. 2.1(b), which looks blurred justifying the incapability of image zooming methods in improving resolution of an image.

2.2 Scheme for super-resolution:


The super-resolution image reconstruction methods consist of three basic components: (i) motion compensation (ii) interpolation and (iii) blur and noise removal, as shown in Fig. 2.2. Motion compensation is used to map the motion from all available low resolution frames to a common reference frame. The motion field can be modeled in terms of motion vectors or as affine transformations. The next stage refers to mapping of motion compensated pixels onto a superresolution grid. The third component is needed to remove the sensor and optical blurring.

Y Y2

e g i s t r a t i o o r M o t i o n E s t i m a t i o n

nI n

Yn

t e n h i r e s g r o

r p o l a t i o t o a g h o l u t i o n i d

n R e s t o r a St i ou - r np e e s r o f o r b l u r i am n a d g e n o i s e r e m o v a l

l u

t i o

Fig. 2.2 Scheme for super-resolution from multi-frame, shifted observations

It is possible to increase spatial resolution in super-resolution technique as multiple low resolution images are available which are obtained from the same scene. These low resolution observations are aliased as well as misregistered with sub-pixel shifts [21]. Fig. 2.3 gives the relationship between low resolution observations and high resolution image. In Fig. 2.3(a) four, 4 X 4 low resolution observations are shown with sub-pixel shift in each frame. A high resolution image of 8 X 8 shown in Fig. 2.3(b) is reconstructed by fusing information present in low resolution frames. If low resolution images are shifted by integer units, then no new information is available to reconstruct the high resolution image.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 7

PROJECT TITLE

2011-12

__________________________________________________________________

(a)

(b)

Fig. 2.3 Relationship between low resolution images and high resolution image

2.3 Super-resolution as an ill posed inverse problem:


Two problems are called inverses of one another if the formulation of each involves all or part of the solution of the other. An example of direct/inverse problem is polynomials.

2.4 Discussion:
Basic understanding of super-resolution concept is given in this chapter. An example is given to show image zooming will not improve resolution. As high frequency components are lost during the low resolution sampling process, resizing of image cannot recover these frequency components. It is possible to reconstruct a high resolution image from available low resolution frames, if they are aliased and sub-pixel shifted. A brief theory of inverse, ill-posed problems is described. Since multi-frame super-resolution does not satisfy Hadamard's condition it is considered to be ill-posed inverse problem.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 8

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 3 Iterative Back Projection method


Super-resolution is generally applied to overcome the limitations of optical imaging systems through the use of image processing techniques. In this chapter, IBP algorithm is discussed. Here super-resolution is presented as a large sparse linear optimization problem. The matrix-vector formulation is used for analysis and implementation is done with standard operations such as convolution, warping and sampling. This has the advantage of speedup in computation. Implementation is done with gradient based optimization techniques.

3.1 Observation model:


When an image is acquired using a camera, there is a natural loss of spatial resolution caused by optical distortions, motion blur, noise that occurs within the sensor and insufficient sensor density as shown in Fig. 3.1. Thus the recorded image suffers from blur, noise and aliasing effects. Main concern of super-resolution is to recover high resolution image from blurred, noisy, aliased low resolution image.

Fig. 3.1 Common image acquisition system (Courtesy: S. C. Park et. al.[4])

Multiframe super-resolution is an inverse problem. The imaging process that results in the

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 9

PROJECT TITLE

2011-12

__________________________________________________________________
observed image sequence represents the direct problem. The observation model can be formulated as follows:

(3.1) Where: xH : HR image XH of size N1 X N2, lexicographically reordered as N1N2 X 1 vector xL(n) : nth LR image of size M1 X M2, lexicographically reordered as M1M2 X 1 vector en : Noise in the nth image, lexicographically reordered as a vector Wn : Geometric warp matrix of size N1N2 X N1N2 Hn : Blurring matrix of size N1N2 X N1N2 D : Decimation matrix of size M1M2 X N1N2

3.2 IBP algorithm:


Irani and Peleg [26] formulated the iterative back projection method which is similar to the back projection used in tomography.

3.3 Implementational issues:


Calculation of gradient in eq. 3.4 involves operations on large matrices which can be computationally very intensive. The matrices W, H, D and their transposes can be implemented using image operations and considerable speed-up can be achieved [28]. (a) Down-sampling: Consider a small image of size 4 X 4 pixels given by
a1 a a= 5 a9 a 13

a2 a3 a4 a6 a7 a8 a 10 a11 a12 a 14 a15 a16

(3.8)

After down-sampling, we get


b=

[ ]
b1 b 2 b3 b 4

(3.9)

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 10

PROJECT TITLE

2011-12

__________________________________________________________________

For down-sampling by a factor of q=2, we need (q2-1=3) additions and one multiplication (by
1 1 = ) per pixel in the lower dimension. The down-sampling operation can be expressed in q2 4

matrix form as b = Da where


1 1 0 D= 4 0 0

1 0 0 0

0 1 0 0

0 1 0 0

1 0 0 0

1 0 0 0

0 1 0 0

0 1 0 0

0 0 1 0

0 0 1 0

0 0 0 1

0 0 0 1

0 0 1 0

0 0 1 0

0 0 0 1

0 0 0 1

(3.10)

(b) Warping: The warping operation is typically performed using bilinear interpolation. Each pixel value in the warped image is calculated from its neighboring pixels using the interpolation coefficients. Assume a 2 X 2 pixels image
a=

[ ]
a1 a2 a3 a4
b1 b 2 b3 b 4

(3.12)

which must be warped by sub-pixel shifts x and y in the x and y directions, respectively. The values of the pixels corresponding to the warped image
b=

[ ]

(3.13)

(c) Blurring: To explain blurring, 1D convolution is taken as an example.

3.4 Discussion:
Goal of super-resolution is to restore high resolution image from degraded and aliased LR images. Degradation is due to motion that occurs during image acquisition, blur caused by optical system and noise added. Aliasing is due to lesser sensor density. An observation model is described which gives relation between high resolution image and low resolution frames. Matrix-vector formulation is used for analysis. However, standard image operations are used for implementation which results in speedup of super-resolution computation.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 11

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 4 Super-resolution for 1-D signal


Images taken by LR imaging system always suffer from degradation and aliasing. Superresolution, in addition to deblurring, denoising also performs dealiasing. Thus it is considered as second generation problem of image restoration. In this chapter dealiasing capability of superresolution is demonstrated on one-dimensional signals using IBP algorithm.

4.1 Observation model for 1-D signal: 4.2 Results for single tone sinusoidal signal:
The algorithm is demonstrated for single tone sinusoidal signal with results in frequency domain. Noise and blur are considered to be Gaussian with zero mean. The initial estimate is chosen to be average of bilinear interpolation of the available low resolution signals. The mean square error (MSE) is calculated as

(4.2) where: XH : High resolution signal, n: Total number of samples, k: Number of iterations
xH

is the super-resolved signal.

4.5 Discussion:
Algorithm was tested on seveal 1-D signals. Dealiasing capability of super-resolution was clearly demonstrated with single tone and dual tone sinusoidal signals. Experiments were conducted under different conditions like ideal situation, high blur and high noise. In ideal situations, reconstruction is very good, but fails with high blur and/or noise demonstrating the sensitivity of the algorithm to noise and blur. _____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 12

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 5 IBP results for images


Dealiasing capability of super-resolution makes it possible to reconstruct high resolution image from aliased LR frames. This is demonstrated on 1-D signals using IBP algorithm in chapter 4. In this chapter, performance of the iterative back projection method is demonstrated on several synthetic images.

5.1 Experimental results:


Experiments were conducted with ideal situation, high blur and high noise. The blur is assumed to be Gaussian and space invariant. Noise is Gaussian with zero mean. LR images were simulated from HR image by warping, blurring and downsampling.

(a)

(b)

(c)

(d)

Fig. 5.1 Results for synthetic image with low blur and noise (a) High resolution image, (b) Low resolution

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 13

PROJECT TITLE

2011-12

__________________________________________________________________
observation, (c) Interpolated image, (d) Reconstructed image.

Case 1: Ideal situation


For simulation, a high quality image of 192 X 256 pixels ( Fig. 5.1(a)) was chosen.

5.2 Results with different initial estimate


Gradient based methods usually do not depend on the initial estimate with which the algorithm starts.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Fig. 5.5 Results for synthetic image with different initial guess (a) High resolution image, (b) Low resolution observation, (c) Initial guess, Reconstructed image after (d) 5 iterations, (e) 50 iterations, (f) 75 iterations, (g) 100 iterations, (h) 125 iterations, (i) 150 iterations

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 14

PROJECT TITLE

2011-12

__________________________________________________________________

5.4 Discussion:
In this chapter, IBP algorithm was tested on several synthetic images under different conditions. Eventhough IBP performs well under ideal conditions, it degrades quickly when observations are highly blurred and noisy. This is due to ill-posedness of super-resolution problem. Also, independence of IBP algorithm was tested with different initial guess. LR observations when shifted with integer amount contain no new information and thus HR image cannot be reconstructed.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 15

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 7 Software description


Iterative back projection algorithm is one of the popular motion based super-resolution algorithms. A high resolution image is reconstructed from available low resolution observations. As high resolution image is unknown, initial guess is required for the simulation of algorithm in eq. 3.7. In this project, we used available LR observations and re-sized using interpolation to have initial guess. The algorithm is implemented on 1-D signal and images using MATLAB functions. Also, in order to implement the algorithm on a DSP processor C code was written. A general flowchart is given in this chapter. Two functions are implemented, project and backproject. (a) project function implements the expression DHnWnxH(m). With initial estimate, xH (m) as input, it simulates LR images as follows: 1. Warp the image with sub-pixel shifts using backward warping 2. Blur the image with Gaussian low pass kernel 3. Decimate the image by a factor q to get an image of the size of LR image. q will be the down-sampling factor (b) Obtain the residual error by subtracting simulated LR images from observed LR images. (c) backproject updates the HR image by adding initial estimate. This operation implements the expression With residual error as input, the function backproject is implemented as follows: 1. Upsample the image to the size of the super-resolved image, by equally spreading the intensity value in lower dimension to q2 pixels in higher dimension. 2. Blur the upsampled image with flipped kernel. 3. Warp the image using forward warping (d) Update the initial estimate (e) Repeat the steps (a) to (d) for required number of iterations

Flowchart for the IBP algorithm:


_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 16

PROJECT TITLE

2011-12

__________________________________________________________________
Start

Initial estimate, Iteration count

Project

Compute residual error between observed and simulated images

Backproject

no

Is count = 0?

yes
Stop

Fig. 7.1 Flow chart for IBP algorithm

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 17

PROJECT TITLE

2011-12

__________________________________________________________________

Chapter 8 Conclusion
Super-resolution is a signal processing technique to obtain high resolution image from observed multiple low-resolution images. The major advantage of this technique is that it costs less and existing low resolution imaging systems can still be utilized. In this report, we considered superresolution of several 1-D signals and synthetic images. Experiments were carried out using a popular iterative back projection algorithm. With this gradient based algorithm speed-up in the computation was achieved as implementations were done using standard image operations. Initially, dealiasing capability of super-resolution algorithm was demonstrated by considering single tone and dual tone sinusoidal signals. When noise and blur were negligible, the reconstructed signal has close match to original. However, with high blur and high noise reconstruction was not good thus proving sensitivity of algorithm to noise and blur. Several experiments were also conducted taking a row of an image as high quality signal. Next, experiments were conducted on synthetic images. Here again under ideal conditions superresolved image is good. But with high blur and high noise, algorithm fails to remove the the degradations completely in the recovered image. Further, convergence of the algorithm was tested with a different initial guess. Demonstration was also done to show that super-resolution fails with integer shifts. In addition, the algorithm using DSP processor was also implemented and demonstrated using code composer studio and DSP kit.

8.1 Suggestions for future work:


Using iterative back projection algorithm, by adding regularization, improvement can be obtained under noisy conditions Computational benefits can be achieved with advanced gradient techniques Fast processing is possible in DSP processor using MATLAB and Simulink.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 18

PROJECT TITLE

2011-12

__________________________________________________________________

BIBLIOGRAPHY:
1. R. C. Gonzalez and R. E. Woods, Digital image processing. Asia: Pearson Education, 2006 2. Sonka, Hlavac, and Boyle, Digital image processing and computer vision, Cengage Learning

3. S. Chaudhuri, Super-resolution imaging. Bostan: Kulwer Academic Publishers, 2001. 4. S. C. Park, M. K. Park, and M. G. Kang, Super-resolution image reconstruction: A technical review, IEEE Signal Processing Magazine, vol. 20, pp. 21-36, May 2003. 5. R. Y. Tsai and T. S. Huang, Multiframe image restoration and registration, Advances in Computer Vision and Image Processing, vol. 1, pp. 317-339, 1984. 6. S. Peleg, D. Keren and L. Schweitzer, Improving image resolution using subpixel motion, Pattern Recognition Letters, 1987. 7. K. V. Suresh, G. Mahesh Kumar, and A. N. Rajagopalan, Super-resolution of license plates in real traffic videos, IEEE Transactions on Intelligent Transportation Systems (ITS), vol. 8, pp. 321 331, 2007. 8. M. I. Chacon, I. Valenzuela, Fast image processing application development scheme for the DSK C6711 using MATLAB and SIMULINK, IEEE-11th Digital Signal Processing Workshop and IEEE Signal Processing Education Workshop, pp. 79 83, 2004. 9. N. Kehtarnavaz, Real-Time Digital Signal Processing - Based on the TMS320C6000, Prentice Hall 10. A. Singh and S. Srinivasan, Digital Signal Processing, Thomson Learning, 2004

11. S. Kim and W. Su, Recursive high-resolution reconstruction of blurred multiframe images, Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing, volume 4, pages 2977 2980, Toronto, Canada, 1991.

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 19

PROJECT TITLE

2011-12

__________________________________________________________________

_____________________________________________________________________________ Electronics & Communication, SIT, Tumkur-3 20