2013 - Gradient-Based Interpolation Method For

Gradient-based interpolation method for
division-of-focal-plane polarimeters
Shengkui Gao and Viktor Gruev*
Dept. of Computer Science and Engineering, Washington University in St. Louis, One Brookings Drive, Saint Louis,
MO, USA 63130
*
vgruev@wustl.edu
Abstract: Recent advancements in nanotechnology and nanofabrication

have allowed for the emergence of the division-of-focal-plane (DoFP)
polarization imaging sensors. These sensors capture polarization properties
of the optical field at every imaging frame. However, the DoFP polarization
imaging sensors suffer from large registration error as well as reduced
spatial-resolution output. These drawbacks can be improved by applying
proper image interpolation methods for the reconstruction of the
polarization results. In this paper, we present a new gradient-based
interpolation method for DoFP polarimeters. The performance of the
proposed interpolation method is evaluated against several previously
published interpolation methods by using visual examples and root mean
square error (RMSE) comparison. We found that the proposed gradient-
based interpolation method can achieve better visual results while
maintaining a lower RMSE than other interpolation methods under various
dynamic ranges of a scene ranging from dim to bright conditions.
©2013 Optical Society of America
OCIS codes: (110.5405) Polarimetric imaging; (120.5410) Polarimetry; (230.5440)
Polarization-selective devices; (260.5430) Polarization.
References and links

1. V. Gruev, R. Perkins, and T. York, “CCD polarization imaging sensor with aluminum nanowire optical filters,”
Opt. Express 18(18), 19087–19094 (2010).
2. R. Perkins and V. Gruev, “Signal-to-noise analysis of Stokes parameters in division of focal plane polarimeters,”
Opt. Express 18(25), 25815–25824 (2010).
3. T. York and V. Gruev, “Characterization of a visible spectrum division-of-focal-plane polarimeter,” Appl. Opt.
51(22), 5392–5400 (2012).
4. M. Kulkarni and V. Gruev, “Integrated spectral-polarization imaging sensor with aluminum nanowire
polarization filters,” Opt. Express 20(21), 22997–23012 (2012).
5. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote
sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006).
6. Y. Liua, T. York, W. Akersa, G. Sudlowa, V. Gruev, and S. Achilefua, “Complementary fluorescence-
polarization microscopy using division-of-focal-plane polarization imaging sensor,” J. Biomed. Opt. 17(11),
116001.1–116001.4 (2012).
7. D. Miyazaki, R. Tan, K. Hara, and K. Ikeuchi, “Polarization-based inverse rendering from a single view,”
Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on, 2, 982–987 (2003).
8. Y. Y. Schechner and N. Karpel, “Recovery of underwater visibility and structure by polarization analysis,” IEEE
J. Oceanic Eng. 30(3), 570–587 (2005).
9. T. V. T. Krishna, C. D. Creusere, and D. G. Voelz, “Passive polarimetric imagery-based material classification
robust to illumination source position and viewpoint,” IEEE Trans. Image Process. 20(1), 288–292 (2011).
10. M. Sarkar, D. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing CMOS
image sensor for material classification,” IEEE Sens. J. 11(8), 1692–1703 (2011).
11. M. Anastasiadou, A. D. Martino, D. Clement, F. Li’ege, B. Laude-Boulesteix, N. Quang, J. Dreyfuss, B. Huynh,
A. Nazac, L. Schwartz, and H. Cohen, “Polarimetric imaging for the diagnosis of cervical cancer,” Phys. Status
Solidi 5(5), 1423–1426 (2008).
12. E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,”
Opt. Spectrosc. 107(6), 884–890 (2009).
13. B. M. Ratliff, C. F. LaCasse, and J. S. Tyo, “Interpolation strategies for reducing IFOV artifacts in microgrid
polarimeter imagery,” Opt. Express 17(11), 9112–9125 (2009).
#177152 - $15.00 USD Received 1 Oct 2012; revised 10 Dec 2012; accepted 24 Dec 2012; published 10 Jan 2013
(C) 2013 OSA 14 January 2013 / Vol. 21, No. 1 / OPTICS EXPRESS 1137
14. B. E. Bayer, “Color imaging array,” U.S. Patent 3,971,065, Jul 20, 1976.
15. D. Zhou, “An edge-directed bicubic interpolation algorithm,” Image and Signal Processing (CISP), 2010 3rd
International Congress on, vol.3, 1186–1189, 16–18 Oct. 2010.
16. A. Giachetti and N. Asuni, “Real-time artifact-free Image upscaling,” IEEE Trans. Image Process. 20(10), 2760–
2768 (2011).
17. S. Gao and V. Gruev, “Image interpolation methods evaluation for division of focal plane polarimeters,” Proc.
SPIE 8012, 80120N, 80120N-10 (2011).
18. S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,” Opt.
Express 19(27), 26161–26173 (2011).
19. X. Xu, M. Kulkarni, A. Nehorai, and V. Gruev, “A correlation-based interpolation algorithm for division-of-
focal-plane polarization sensors,” Proc. SPIE 8364, 83640L, 83640L-8 (2012).
20. S. Gao and V. Gruev, “Gradient based interpolation for division of focal plane polarization imaging sensors,”
Circuits and Systems (ISCAS), 2012 IEEE International Symposium on, 1855–1858, 20–23 May 2012.
21. Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “Polarization-based vision through haze,” in ACM
SIGGRAPH ASIA 2008 courses, New York, NY, USA, 71:1–71:15 (2008).
22. T. W. Cronin, E. J. Warrant, and B. Greiner, “Celestial polarization patterns during twilight,” Appl. Opt. 45(22),
5582–5589 (2006).
23. J. J. Peltzer, P. D. Flammer, T. E. Furtak, R. T. Collins, and R. E. Hollingsworth, “Ultra-high extinction ratio
micropolarizers using plasmonic lenses,” Opt. Express 19(19), 18072–18079 (2011).
24. Y. Zhao, M. A. Belkin, and A. Alù, “Twisted optical metamaterials for planarized, ultrathin, broadband circular
polarizers,” Nat. Commun. 3(870), (2012).
25. Y. Zhao and A. Alù, “Manipulating light polarization with ultrathin plasmonic metasurfaces,” Phys. Rev. B
84(20), 205428 (2011).
26. R. Keys, “Cubic convolution interpolation for digital image processing,” Acoustics, Speech and Signal
Processing, IEEE Transactions on 29(6), 1153–1160 (1981).
27. J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698
(1986).
28. A. C. Neville and B. M. Luke, “Form optical activity in crustacean cuticle,” J. Insect Physiol. 17(3), 519–526
(1971).
29. Prof. Justin Marshall, Sensory Neurobiology Group, University of Queensland, Brisbane Queensland 4072,
Australia, (personal communication, 2012).
1. Introduction
1.1 Background
There are three fundamental properties of light, and they are intensity, wavelength and
polarization. The first two properties of light are encoded by our visual system as brightness
and color, and the current imaging technology aims to replicate these two properties in high
resolution with low spatial and temporal noise sensors. However, the third property of light,
i.e. polarization, has been largely ignored by the imaging technology in part by the fact that
the human eye is polarization insensitive.
Due to the recent advancements in nanotechnology and nanofabrication, polarization
imaging sensors for the visible spectrum known as the division-of-focal-plane (DoFP)
polarimeters have emerged on the scene [1–5]. These sensors monolithically integrate and
combine pixelated polarization filters with an array of imaging elements. The pixelated
polarization filters are composed of metallic nanowires, which are typically oriented at 0°,
45°, 90° and 135°. An example of this type of polarization imaging sensor is presented in Fig.
1. One of the main advantages of division-of-focal-plane sensors is the capability of capturing
polarization information at every frame. The polarization information captured by this class
of sensors can be used to extract various parameters from an imaged scene, such as
microscopy for tumor margin detection [6], 3-D shape reconstruction from a single image [7],
underwater imaging [8], material classification [9,10] and cancer diagnosis [11,12].
Fig. 1. Block diagram of division-of-focal-plane polarization imaging sensor. The array of
charge coupled device (CCD) imaging elements is covered with four pixelated linear
polarization filters oriented at 0°, 45°, 90° and 135°
There are two main disadvantages of the DoFP polarization sensors and they are: loss of
spatial resolution and inaccuracy of the captured polarization information. Due to the
sampling of the imaged scene with four spatially distributed pixelated polarization filters, the
instantaneous field of view for the neighboring pixels in a 2-by-2 super-pixel configuration
can be different from each other. Hence, the computed polarization information, such as the
first three Stokes parameters, angle and degree of linear polarization, will contain an error
from the true polarization signature of a target. Furthermore, the four types of polarization
pixels which are distributed throughout the imaging array sub-sample the imaged
environment by a factor of four and lead to loss of spatial resolution [13]. These two
shortcomings need to be addressed in order to take full advantage of the real-time imaging
capabilities of division-of-focal-plane polarization sensors.
Similar problems were encountered in color imaging sensors, when the Bayer pattern for
pixelated color filters was introduced in the 1970s [14]. In order to recover the loss of spatial
resolution in color sensors and improve the accuracy of the captured color information,
various image interpolation algorithms have been developed in the last 30 years. Recent color
interpolation methods have focused on developing computationally efficient algorithms for
embedded hardware with high reconstruction accuracy [15, 16]. Although non-linear based
interpolation methods have been widely used in the color domain, these algorithms do not
directly translate to an implementation in the polarization domain due to the inherent
difference of these two imaging modalities. For example, edge detection in the color domain
is performed on the green channel in an image due to the high spatial resolution of this
channel. Polarization domain does not favor any particular polarization orientation and
therefore requires development of novel gradient-based interpolation methods tailored for this
modality.
Interpolation algorithms for DoFP polarization sensors are still at the infancy stage, and to
date, only few interpolation algorithms have been implemented and evaluated, such as
bilinear [13], bicubic, bicubic spline interpolation [17, 18] and correlation-based interpolation
[19]. These interpolation methods for division-of-focal-plane imaging sensors assume that
information across the imaging array varies relatively smoothly and that there are no
discontinuities in the low resolution image. This assumption fails in typically imaged
scenarios, where multiple objects are imaged against a background. The discontinuity in
either polarization or intensity domain, which is generated at the boundary between an object
and the background, will generate false and often strong polarization signatures at the
boundary. In other words, bilinear and bicubic interpolation methods are essentially low pass
filters and smooth out the intensity information recorded by the four feature angle images.
Due to the smoothing of the intensity images, artifacts at the edges of objects in the angle and
degree of polarization image will be generated. These artifacts are undesirable for most
imaging scenarios and should be suppressed through proper image processing techniques.
In this paper, we introduce a new gradient-based interpolation method with adaptive
threshold selection for DoFP polarization sensors. Preliminary results of the proposed
interpolation method are presented in [20]. In this paper, we have expanded the previously
published results and have included additional theoretical analysis and measurements for the
interpolation method. Specifically, we have included detailed theoretical overview of the
proposed algorithm, performance evaluation of the proposed method as a function of scene
brightness and included interpolation results implemented on a division-of-focal-plane sensor
operating in the visible spectrum. The performance of the proposed interpolation algorithm is
compared against the performance of the previously published interpolation methods, such as
bilinear, bicubic spline interpolation and bicubic convolution [13, 17, 18]. Correlation-based
interpolation method for polarization images has been proposed by Xu et al. [19]. This
interpolation method has been evaluated on synthetic images and can achieve higher accuracy
compared to bilinear and bicubic interpolation methods in the reconstructed high resolution
images [19]. Unfortunately this interpolation method only applies to specific cases of
interpolation on synthetic images and cannot be implemented on real-life images as presented
in [19]. Hence, the performance of this interpolation algorithm is not compared against our
proposed gradient-based interpolation method.
Based on the evaluation results, the proposed algorithm outperforms previously published
interpolation algorithms for DoFP sensors in terms of highest accuracy of the recovered high
resolution images. The effects of the scene dynamic range on the accuracy of the interpolated
polarization information are also explored in this paper. The motivation for this evaluation
comes from the fact that many polarization imaging applications are performed under poor
light conditions such as haze imaging [21] and twilight imaging [22]. The dynamic range of a
scene is relatively poor in these examples and the accuracy of the interpolation algorithms is
of a particular interest.
1.2 Linear polarization computation
The DoFP imaging sensors capture both intensity and polarization information of an imaged
scene at every frame. In terms of polarization properties of an imaged scene, typically two
polarization sub-properties are of interest and they are the angle of polarization (AoP) and the
degree of linear polarization (DoLP). The intensity, DoLP and AoP are computed via Eqs. (1)
through (3) respectively:
Intensity = (1 2 )( I (0o ) + I (45o ) + I (90o ) + I (135o )) (1)
( I (0 ) − I (90 ) ) + ( I (45 ) − I (135 ) )

2 2
DoLP = o o o o
Intensity (2)
(
AoP = (1 2)arc tan ( I (45o ) − I (135o ) ) ( I (0 ) − I (90 ) ) )
o o
(3)
o
where I(x ) is the intensity of the light wave filtered with a linear polarization filter in the
x degree orientation. Following Eqs. (1) through (3), an imaging sensor capable of recovering
polarization information from a scene has to sample the imaged environment with four linear
polarization filters offset by 45°. In the case of DoFP polarization sensors, the linear
polarization filters are pixelated and embedded at the focal plane of the imaging sensor (see
the block diagram in Fig. 1). The fourth Stokes parameter is not captured with the sensor
described in Fig. 1 due to the lack of quarter-wave retarder embedded at the pixel level in the
imaging array. These filters can be incorporated in future versions of DoFP polarization
imaging sensors by adding pixelated plasmonic quarter-wave retarders in combination with
linear polarization filters [23–25].
The structure of this paper is as follows: in Section II the bicubic convolution and the
gradient-based interpolation are presented; in Section III the experimental setup is introduced.
In Section IV both visual and root mean square error comparisons are provided on a set of
polarization images and for several different interpolation methods. In this section, adaptive
threshold selection and the scene’s dynamic range influence on the proposed interpolation
method are also discussed. Interpolation results obtained from a DoFP imager are presented at
the end of this section. Concluding remarks are provided at the end of the paper.
2. Gradient-based interpolation
In this section, the bicubic convolution interpolation method is first briefly revisited. An
approximation of the bicubic interpolation method is presented followed by an overview of
the proposed gradient-based interpolation method. The proposed gradient-based interpolation
method employs both the approximated bicubic interpolation method and bilinear
interpolation in order to recover spatial resolution with high accuracy.
2.1 Bicubic convolution interpolation method
The commonly-used bicubic interpolation method attempts to fit a surface between four
corner points of the low resolution image using a third-order polynomial function. In order to
compute the intensity value of the target pixel via the bicubic interpolation method, the spatial
derivatives in the horizontal, vertical and diagonal directions at the four corner points are
initially computed. A total of twelve spatial derivatives are computed and combined with
intensity values from the four corner points in order to compute the interpolated pixel value.
Hence, the bicubic interpolation algorithm is very computational intensive algorithm and
difficult to be realized in real-time on high resolution images.
The bicubic convolution method is an approximation of the bicubic interpolation method
and is a more computationally-efficient algorithm than the bicubic interpolation at the cost of
lower reconstruction accuracy [26]. The bicubic convolution process is first computed in the
x-direction followed by convolution in the y-direction. The raw image, as recorded by the
DoFP sensor, is composed of four sub-images sampled with four linear polarization filters
respectively. Each one of the four sub-images has a quarter of the total resolution of the DoFP
imaging array. For example, if the DoFP sensor is composed of 1000 by 1000 pixels, the sub-
image corresponding to the pixels covered with 0° linear polarization filters contains 500 by
500 pixels. The sub-images corresponding to the pixels covered with 45°, 90° and 135° also
contain 500 by 500 pixels respectively. These four sub-images are referred as low resolution
images and the aim of the interpolation algorithms is to recover the full 1000 by 1000 pixels
resolution for all four sub-images with the highest accuracy.
The interpolated pixel I(x,y) located between the low-resolution pixels f(x,y) and f(x + 1,y)
is computed via the convolution process described by Eq. (4).
2
I ( x, y ) =  W ( s ) f ( x + s , y )
s =−1 (4)
for = [1,N - 1] and y = [1,M]
where the bicubic convolution kernel, W(s), can be described via Eq. (5):
 3 / 2 s 3 − 5 / 2 s 2 + 1 0 < s ≤1
 3 2
W ( s ) =  −1 / 2  s + 5 / 2  s − 4  s + 2 1 < s ≤ 2 (5)
 0 2< s

In Eqs. (4) and (5), N is the number of pixels in the x-direction; M is the number of pixels
in the y-direction; s is the relative location of a pixel in the convolution kernel. Expanding Eq.
(4) into a matrix form, the bicubic convolution interpolation is computed via Eq. (6):
 1 2 ′  0 2 0 0   f ( x − 1, y ) 
 1 4   −1 0 1 0   f ( x, y )  x = 1 N − 1
I ( x, y ) =       (6)
 1 8   2 −5 4 −1  f ( x + 1, y )  y = 1 M
    
1 16  −1 3 −3 1   f ( x + 2, y ) 
Special boundary conditions are applied to the first and last pixel of each row in the
image. The conditions are described by Eq. (7).
 f (0, y ) = 3 f (1, y ) − 3 f (2, y ) + f (3, y )
 (7)
 f ( N + 1, y ) = 3 f ( N , y ) − 3 f ( N − 1, y ) + f ( N − 2, y )
Once the image is convolved in the x-direction, the same convolution method is repeated
in the y-direction of the image in order to complete the interpolation process.
2.2 Gradient-based interpolation method
In this paper, we propose a gradient-based interpolation method in order to improve the
quality of polarization information captured by a DoFP polarization sensor. The fundamental
idea behind the gradient-based algorithm is the following: if an edge is encountered in the
scene, then interpolation is performed along the edge and not across the edge. In the proposed
implementation, bicubic convolution interpolation is performed along an edge in the image.
Bilinear interpolation method is implemented in the smooth areas of the image, i.e. in the
areas where no edges have been detected. For low spatial frequency features, both bilinear
and bicubic interpolation results have similar error reconstruction and modulation transfer
function (MTF) response [18]. Bicubic interpolation outperforms bilinear interpolation in
terms of RMSE and MTF for medium and high spatial frequency features. Due to the lower
computation complexity of executing bilinear interpolation over bicubic convolution and the
similar accuracy in the reconstructed images between both methods, bilinear interpolation is
employed on the smooth features of the image.
In order to implement this algorithm, the boundary discontinuities (or edges) in an image
are initially computed. An edge in an image is defined as a place where the spatial gradient
exceeds a given threshold, and typically identifies an area in the image where one object ends
and another one starts. For example, in color images, a high gradient exists between two
objects with different colors or intensities. In polarization images, a high gradient exists
between objects with different intensities, angles of polarization or degrees of linear
polarization. These gradients imply that edges also exist in the 4 composite images, i.e. in the
0°, 45°, 90° and 135° images.
In the gradient-based interpolation method, the bicubic convolution and the bilinear
interpolation methods are expanded to incorporate gradient selectivity features. To this extent,
the gradients across the vertical, horizontal and two diagonal directions in a 7-by-7 low-
resolution pixel neighborhood on all the four images (0°, 45°, 90° and 135°) are computed
using Eq. (8).
 D 0o =   I (i, j ) − I (i, j − 2)
 i = 2,4,6 j = 3,5,7

 D 45o =   I (i, j ) − I (i + 2, j − 2)
 i =1,3,5 j = 3,5,7
 (8)
 D90o =   I (i, j ) − I (i − 2, j )
 i = 3,5,7 j = 2,4,6
 D o =   I (i, j ) − I (i + 2, j + 2)
 135 i =1,3,5 j =1,3,5
In the above equations, D0° is the gradient in the horizontal direction; D90° is the gradient
in the vertical direction and so on. Figure 2 illustrates how the gradient in the 135° diagonal
and horizontal direction are computed. The black pixel in Fig. 2 denotes the target pixel of the
interpolation computation.
An optimal gradient detection filter should satisfy the following three goals: an accurate
detection selectivity of edges, an accurate localization of edges and minimum response to
false edges [27]. Following the above mention criteria, a small convolution kernel (i.e. 2 by 2
pixels window) will be very sensitive to image noise and will result in many false-negative
edges. A large convolution kernel will smooth out the image, which will suppress many edges
that should be detected. We have selected an intermediate size for the convolution kernel (i.e.
7 by 7 pixels window), which provides a balance between suppressing many false edges due
to noise as well as keep the image fidelity in order to detect the desirable edges of objects.
Depending on the relative positions between known and unknown pixels, the interpolation
is implemented first on the pixels with known value in the diagonal directions, followed by
interpolation on the pixels with known values in its horizontal or vertical direction. As shown
in the following steps:
1) If the ratio of the gradient in the 45° direction over the gradient in the 135° direction
exceed a threshold, i.e. D45°/ D135°>Tthreshold which indicates an edge along the 135° diagonal
direction, the bicubic convolution method is applied to the target pixel along the 135°
diagonal direction. Similarly, if the ratio of the gradient in the 135° direction over the gradient
in the 45° direction exceed a threshold, i.e. D135° / D45° > Tthreshold, the bicubic convolution
method is applied to the target pixel along the 45° diagonal direction. In all other cases, the
intensity value of the target pixel is the average of the intensity values of its four adjacent
pixels in the diagonal directions.
2) If the ratio of the gradient in the 0 degree direction over the gradient in the 90 degree
direction exceed a threshold, i.e. D0°/D90°>Tthreshold which indicates an edge along the vertical
direction, the bicubic convolution is applied to the target pixel along the vertical direction. If
the ratio of the gradient in the 90 degree direction over the gradient in the 0 degree direction
exceed a threshold, i.e. D90°/D0°>Tthreshold, bicubic convolution is applied to the target pixel
along the horizontal direction. In all other cases, the intensity value of the target pixel is the
average of its four adjacent pixels in horizontal and vertical directions.
The Tthreshold value is decided using histogram distribution of the derivative ratios in
diagonal direction (D45°/ D135°) across the imaging scenes for all four angles (0°, 45°, 90° and
135°). From our experimental results, we have determined that setting the Tthreshold value to be
between 55% and 65% of the gradient’s ratio cumulated distribution function (CDF), yields to
best reconstruction results as explained in the results section (see Sec. 4.3).
Fig. 2. (a) 135° direction gradient example. (b) 0° direction gradient example. Note: the blue
pixels are pixels of known value, the black one is the target pixel, the gray ones are the pixels
with same type as the target pixel, and the white ones are pixels of unknown value.
The computational complexity of an interpolation method is an important consideration

for real-time implementation. The proposed gradient-based interpolation is computational
more intensive compared to bilinear and bicubic convolution method. For example, the edge
convolution filter requires 2 multiplications, 34 additions and one thresholding operation per
pixel. Bicubic convolution method requires 24 multiplications, 23 additions and bilinear
interpolation requires 4 multiplications and 3 additions. The overall computational
complexity of the gradient-based method will depend on the spatial features of the imaged
scene, while bilinear and bicubic interpolation will have a deterministic number of
computational steps.
3. Experimental setup
In order to evaluate the accuracy of different interpolation methods, the true high-resolution
polarization image must be known a priori. Division-of-focal-plane (DoFP) polarization
imaging sensors can only generate low-resolution images. Thus, the performance of the
interpolation algorithms cannot be compared against the true polarization signature since it is
not known a priori. In order to circumvent this problem, a set of images are acquired by using
a gray-scale CCD imaging sensor (Kodak KAI-4022, 40dB SNR, 60dB Dynamic Range)
together with a linear-polarized filter rotation stage (Thorlabs NR360s, 5 arcmin accuracy)
placed in front of the sensor. The linear-polarized filter is rotated at 0°, 45°, 90° and 135°
orientation thus obtaining four co-registered high-resolution images. These four images are
regarded as the “true” high-resolution images since they sample the optical field at every
pixel with four different linear polarization filters offset by 45 degrees.
Next, the four “true” high-resolution images are decimated by following the sampling
pattern of the DoFP polarization imaging sensor shown in Fig. 1. Hence, four low resolution
images at 0°, 45°, 90° and 135° feature angles are obtained and essentially simulate an image
that would be acquired from a DoFP sensor. The four low resolution images are interpolated
via different methods and high resolution images are computed. The final high resolution
interpolated images are compared against the true high-resolution images that were originally
recorded. This experimental set-up aims to provide fair comparison of the reconstruction error
between bilinear, bicubic and gradient-based interpolation methods.
Fig. 3. The true high-resolution image (a) muse-Intensity, (b) muse-DoLP, (c) muse-AoP, (a)
soldier-Intensity, (b) soldier -DoLP, (c) soldier –AoP.
The four high resolution images are recorded with a gray-scale camera with a rotating
linear polarization filter and might contain alignment errors in the optical set-up. The optical
misalignment in the image acquisition set-up will generate errors in the acquired polarization
data. These errors are not critical for our set of experiments since the interpolation algorithms
are implemented on the decimated images computed from the original high resolution images.
The interpolated images are compared against the original high resolution images in order to
estimate the accuracy of the method. Hence, any optical misalignment errors would equally
affect the original high resolution images as well as the interpolated images.
In Figs. 3(a)-3(c), the intensity, degree of linear polarization (DoLP) and angle of
polarization (AoP) results of a still toy muse are presented (Integration time: 100 msec). This
figure presents the “true” high-resolution images and will be used to compare the accuracy of
the interpolated images. The polarization sub-properties, DoLP and AoP, are mapped to
grayscale and false color scales respectively (Figs. 3(b), 3(c)). A second example of a toy
soldier scene is presented in Figs. 3(d)-3(f), which is used to verify the validation of the
adaptive threshold selection (Sec. 4.3).
4. Experimental results
In this section, visual comparison and root mean square error (RMSE) analysis are applied to
the first test image in order to evaluate the performance of the different interpolation methods.
Furthermore, the adaptive threshold selection of the proposed gradient-based interpolation
method and scene’s dynamic range impacts on the interpolated results are presented in this
section. Finally, the interpolated image results from an existing DoFP polarization imager are
shown for further comparison.
4.1 Image visual comparison
A small region in toy muse images shown in Figs. 3(a)-3(c), depicted by a white square, is
used for visual evaluation of the different interpolation algorithms. The region in the white
box is expanded and presented in Fig. 4.
Fig. 4. Comparison of different interpolation methods on the intensity, DoLP and AoP. (a)
True polarization, (b) bilinear interpolation, (c) bicubic spline interpolation, (d) bicubic
convolution and (e) gradient-based interpolation.
The set of images in Fig. 4 are organized as follows: the first column of images presents
the intensity images; the second column of images presents the degree of linear polarization
images and the third column presents the angle of linear polarization images. The first row of
images (Fig. 4(a)) presents the true high-resolution polarization images and are used to
visually compare the reconstruction accuracy of the different interpolation methods presented
in the following four rows.
The images in Fig. 4(b) are obtained via bilinear interpolation and show strong pixilation
effects in the intensity, DoLP and AoP images. In all three images, the details of the muse's
horn are pixelated; the edges (marked as A, B and C) and the background are accentuated and
show strong “toothed artifacts” due to the large error introduced by the bilinear interpolation.
The images presented in Figs. 4(c) and 4(d) are obtained via the bicubic spline and bicubic
convolution interpolation methods respectively. The two bicubic-based interpolation methods
show similar results. In these sets of images, the details of the muse's horn in both DoLP and
AoP are recovered with higher accuracy than with bilinear interpolation. Pixilation effects are
visibly reduced using bicubic interpolation when compared to bilinear technique. However,
the DoLP and AoP images present visible “ripple effect”, especially in the edge area (marked
as A, B and C) between the muse's horn and the background.
The images presented in Fig. 4(e) are the results of gradient-based interpolation (Tthreshold:
60% CDF). This set of images is well-recovered and closely resembles the true polarization
images. There are no large pixilation artifacts observed in any of the three polarization
images. Hence, the gradient-based interpolation method visually recovers a high-resolution
image with the highest accuracy.
4.2 RMSE comparison
In order to provide a numeric comparison of the accuracy of the different interpolation
methods, the root mean square error (RMSE) evaluating method is applied and is shown in
Eq. (9):
1
  (Oc (i, j ) − I c (i, j ) )
2
RMSE = (9)
MN 1≤i ≤ M 1≤ j ≤ N
where Oc(i, j) is the true value of the target pixel, i.e. the image presented in Fig. 4(a), Ic(i,
j) is the interpolated intensity value of the target pixel, M and N represent the number of rows
and columns in the image array respectively.
Table 1. The RMSE performance comparison for toy muse
Bicubic Gradient-based
Bilinear Bicubic Spline
Convolution Interpolation
Intensity 2.40E + 01 (1.48) 1.80E + 01 (1.11) 1.90E + 01 (1.17) 1.62E + 01 (1)
DoLP 2.00E-02 (1.23) 1.94E-02 (1.19) 1.83E-02 (1.12) 1.63E-02 (1)
AoP 6.79E + 00 (1.08) 6.59E + 00 (1.04) 6.42E + 00 (1.02) 6.31E + 00 (1)
I(0°) 2.04E + 01 (1.42) 1.69E + 01 (1.17) 1.73E + 01 (1.20) 1.44E + 01 (1)
I(45°) 1.97E + 01 (1.41) 1.66E + 01 (1.19) 1.69E + 01 (1.21) 1.40E + 01 (1)
I(90°) 1.94E + 01 (1.37) 1.66E + 01 (1.17) 1.69E + 01 (1.19) 1.42E + 01 (1)
I(135°) 1.92E + 01 (1.40) 1.61E + 01 (1.18) 1.64E + 01 (1.20) 1.37E + 01 (1)
The values inside the brackets are the RMSE ratio of the corresponding interpolation to gradient-based interpolation
For the toy muse test images, the RMSE results for the different interpolation methods are
shown in Table 1. The minimum RMSE for I(0°), I(45°), I(90°), I(135°), intensity, DoLP and
AoP images is obtained via the gradient-based interpolation method. The bilinear
interpolation method introduces the largest error for all the comparisons, while the bicubic
spline and bicubic convolution interpolation methods showed similar error performance with
the latter one being computationally efficient.
4.3 Adaptive threshold selection
Accurate threshold selection for discerning edges in an image is a key part for most edge-
detection methods. In order to determine an edge in an image, most algorithms use an
experimentally determined edge-threshold and assign it as a constant for a set of images.
Although this method is computationally efficient, it does not yield to a good edge detection
results as edge-thresholds tend to vary between images.
In the proposed gradient-based interpolation method, the threshold is determined using the
cumulated distribution function (CDF) of the ratio between two orthogonal spatial
derivatives, i.e. D0°/D90° or D45°/D135°., for the intensity image of each angle separately, and
one single threshold value is assigned to each intensity image by assuming the spatial
derivatives in different directions have the same histogram distribution. The edge-thresholds
are determined from the CDF by selecting different percentile of the function. For example,
selecting the threshold value to be 100% of the CDF effectively suppresses edge detection
and yields the same RMSE as a bilinear method. If the threshold value is set to 30% or lower
of the CDF, many false positive edges can be selected which will lead to RMSE similar to the
bicubic convolution. If the edge-threshold is set to 0% of the CDF, all areas in the image will
be defined as edges and bicubic convolution interpolation will be applied across the entire
image.
In order to determine a well-suited threshold value, the RMSE of the DoLP and AoP
image are computed for different threshold values. Since both show the similar trend, only
AoP results are shown in Fig. 5 to keep the briefness.
Fig. 5. Different CDF threshold selection (a) the normalized AoP RMSE of the toy muse, (b)
the normalized AoP RMSE of the toy soldier .
The normalized AoP RMSE (the ratio of the RMSE to the maximum value) of the muse
image is evaluated for five different integration times of the imaging sensor in order to
emulate different dynamic ranges in a scene and is presented in Fig. 5(a). For example, low
integration times allow for few photons to be registered by the array of photodiodes and the
dynamic range of a scene is very low. Long integration time allow for higher number of
photons to be collected by the array of photodiodes and hence have a higher dynamic range of
the scene (see Fig. 7). The normalized RMSE of the AoP image for the toy soldier image as a
function of the edge-threshold is shown in Fig. 5(b). The RMSE error for both set of images
follow similar trend with respect to the edge-threshold value.
For all five different integration times, the minimum RMSE is obtained when the
threshold value is set between 55% and 65% of the CDF. Hence, the threshold for computing
edges in a scene is set to 60% of the CDF. This rule of thumb is employed on several different
sample images that we have collected and consistent results for minimum RMSE are obtained
for each image. Therefore, setting the edge-detection threshold between 55% and 65% of the
CDF is a good guideline for any image.
4.4 Dynamic range impact on interpolation
The DoFP polarization sensors are used to image scenes with various dynamic ranges,
ranging from very dim scenes to very bright ones. Hence, the accuracy of the interpolated
polarization information as a function of a scene’s dynamic range needs to be closely
evaluated. In this section, a toy muse scene, which is presented in Figs. 3(a)-3(c), is imaged
under different integration times. The integration time of the imaging sensor is swept between
0.5 msec and 120 msec. Note, similar results are obtained when the aperture of the sensor is
modulated or the illumination of the scene is varied between experiments. For briefness of the
paper, we only present the data collected under different integration periods.
Fig. 6. The normalized RMSE results (a) Intensity, (b) DoLP, and (c)AoP; The normalized
STD of RMSE (d) Intensity, (e) DoLP, (f) AoP, for different interpolation methods by
scanning integration time.
The normalized RMSE (the ratio of the RMSE to the maximum value) results for the
intensity, DoLP and AoP as a function of integration time are shown in Figs. 6(a)-6(c)
respectively. Figures 6(d)-6(f) present the corresponding normalized standard deviation
(STD) of the RMSE for the intensity, DoLP and AoP respectively. The RMSE and STD
figures are computed across all pixels in the imaging array. The STD plot follows similar
trend as the RMSE plots presented in Figs. 6(a)-6(c). Furthermore, the gradient-based
interpolation method yields the smallest variations in the error measurements when compared
to bilinear and bicubic interpolation across different integration times.
Fig. 7. Image results from the sensor under different integration time. (a) 0.5 msec, (b) 4 msec,
(c) 12 msec, (d) 20 msec, (e) 40 msec and (f) 100 msec.
Sample images of the intensity, DoLP and AoP as a function of different integration times
are presented in Fig. 7. The images are collected with a normal camera with linear dynamic
range of 60dB, i.e. high dynamic range camera is not used for collecting data. The images
presented in Fig. 7(a) are collected with integration time of 0.5 msec. In this set of images, the
intensity image does not provide visible contrast to the object against the background and the
dynamic range of the scene, defined as the ratio of the difference between the maximum
intensity and the minimum intensity pixel within the scene to the maximum allowed intensity
of the pixel, is 0.5%. The dynamic range of the scene is increased to 3% in the second row of
images and steadily increases in the rest of the images. The dynamic range of the scene for
the intensity image in the last row is 70%. Based on the results shown in Fig. 7, the DoLP and
AoP always provide good contrast and details of the target in a wide intensity range (any
integration time larger than 4msec), while having inaccurate information for small dynamic
range of the scene.
Figure 6(a) shows the error in the intensity image for different interpolation methods. The
RMSE linearly increases with the scene brightness, i.e. integration time of the imaging
sensor. As the integration time increases, the intensity values in an image also increase. The
interpolation methods attempt to compute a missing pixel value and the error will be larger
for brighter scenes. For dim scenes, the intensity values are low and hence the interpolation
errors will be naturally lower. For bright scenes, the intensity values are high and the
interpolation errors will be accordingly higher than the errors from dim scenes. Another
important observation can be made is that the gradient-based interpolation method has the
lowest error compared to the other two interpolation methods.
Figures 6(b) and 6(c) present the normalized RMSE for the degree of linear polarization
and angle of polarization images as a function of the integration time respectively. The error
in the angle and degree of linear polarization is constant for majority of the different
integration times. This can be explained by observing Eqs. (2) and (3). The angle and degree
of linear polarization equations do not depend on the intensity measurements of the target.
Hence, the errors introduced during the interpolation step in the image processing algorithm
are going to be constant for different integration times, i.e. for different dynamic ranges of the
scene. For low light conditions, the reconstruction errors increase due to the fact that the
signal to noise ratio (SNR) of the imaging sensor is low. Hence, the interpolated value will
have lower SNR or higher temporal variations. These temporal variations will lead to higher
inaccuracies in the reconstructed values of the raw pixel values. From Fig. 6, it can be
observed that the proposed gradient-based interpolation method outperforms bilinear and
bicubic interpolation method under both short and long integration times of the imaging
sensor.
Figures 6(d) through 6(f) present the corresponding normalized standard deviation (STD)
of the RMSE for the intensity, DoLP and AoP respectively. The RMSE and STD figures are
computed across all pixels in the imaging array. The STD plot follows similar trend as the
RMSE plots presented in Figs. 6(a)-6(c). Furthermore, the gradient based-interpolation
method yields the smallest variations in the error measurements when compared to bilinear
and bicubic interpolation across different integration times.
4.5 Interpolation results of DoFP polarization imager
The three interpolation methods (bilinear, bicubic-spline and gradient-based) are applied to
the image output of a high-resolution DoFP polarization imaging sensor [1], so that their
reconstruction performance can be evaluated. The division of focal plane sensor is inserted in
an underwater casing and real-life images of a lobster in the natural habitat have been
recorded. The lobster images are presented in Fig. 8. Since the camera is submerged
underwater, there is not specular reflection from the surface of water in this imaging set-up.
Figure 8 shows the interpolated results of a lobster body recorded underwater. Figure 8(a)
shows the bilinear interpolation results. In these set of images, strong horizontal and vertical
artifacts can be observed on the lobster’s leg and tail part in both DoLP and AoP images.
Figure 8(b) shows the bicubic interpolation results and many of the artifacts observed in the
bilinear interpolation have been suppressed. Figure 8(c) shows the gradient-based
interpolation results. In this set of images, many of the edge artifacts that caused strong
polarization signatures in the bilinear and bicubic interpolation are suppressed with the
gradient-based interpolation method.
Examining the above results for each interpolation method, the proposed gradient-based
interpolation method presents the best visual results, and eliminates most of the noise and
artifacts, which could be misinterpreted as “real” polarization information. It has been
reported in the literature that lobsters, do not have polarization information between the shell
ridges [28]. Furthermore, polarization information has not been observed between the shell
ridges when the lobster has been imaged with division of time imaging sensors [29]. The
divisions of time polarimeters produce accurate polarization information at the edges of
stationary objects. The polarization information at the edges of the shell ridges, when
recorded with division of focal plane polarimeters, is incorrectly computed via the bilinear
and bicubic interpolation method and is suppressed via our gradient-based method.
Fig. 8. Interpolated results of DoFP imager on the intensity, DoLP and AoP. (a) bilinear
interpolation, (b) bicubic spline interpolation and (c) gradient-based interpolation.
4. Conclusion
In this paper, we have presented a new gradient-based interpolation method for the division-
of-focal-plane (DoFP) polarization imaging sensor. The performance of the proposed
gradient-based interpolation method is compared against the performance of several other
interpolation methods in terms of visual testing and RMSE comparison. According to the
results, the adaptive gradient-based interpolation method outperforms bilinear and bicubic
interpolation methods using both visual quantitative evaluation as well as qualitative
evaluation using an RMSE method.
The improvements in the reconstruction accuracy using the proposed gradient-based
interpolation method are achieved by applying bicubic convolution interpolation to the edge
area and bilinear interpolation the non-edge areas. The original discontinuity within the edge
area can be well reconstructed, while the smoothness within the non-edge area can also be
maintained. The gradient-based interpolation algorithm outperforms bilinear and bicubic
interpolation algorithms under various dynamic range of a scene ranging from dim to bright
conditions, and it could also bring a large improvement to the output quality of the real DoFP
polarization imager.
Acknowledgments
This work was supported by National Science Foundation grant number OCE-1130897, and
Air Force Office of Scientific Research grant numbers FA9550-10-1-0121 and FA9550-12-1-
0321.

2013 - Gradient-Based Interpolation Method For

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2013 - Gradient-Based Interpolation Method For

Uploaded by

Copyright:

Available Formats

Gradient-based interpolation method for

Abstract: Recent advancements in nanotechnology and nanofabrication

References and links

( I (0 ) − I (90 ) ) + ( I (45 ) − I (135 ) )

The computational complexity of an interpolation method is an important consideration

You might also like