A Novel Smoothness-Based Interpolation Algorithm

for Division of Focal Plane Polarimeters

Jieyun Zhang1, Wenbin Ye1 , Ashfaq Ahmed2, Zhurui Qiu3 , Yuan Cao1 and Xiaojin Zhao1,4,∗
1 College of Electronic Science and Technology, Shenzhen University, Shenzhen, China
2 Department of Bioengineering, the Hong Kong University of Science and Technology, Hong Kong
3 Chenghan International School, Shenzhen, China
4 American University of Ras Al-Khaimah, Ras Al-Khaimah, U.A.E.
∗ Email: eexjzhao@szu.edu.cn

CMOS Image Sensor Column Selection

Abstract—In this paper, we present a novel smoothness-
based interpolation algorithm for the division of focal plane 0 0 0 0 1 1

CMOS Image Sensor Row Selection

polarimeters (DoFP). By calculating the divided blocks’ variance
that represents their local smoothness, the proposed algorithm

1 1 0 0 0 0
well-balances between the traditional bilinear and bicubic inter- micropolarizer for 0r linearly
0º Micro-polarizer
polarized component
polation algorithms. In addition, compared with the previously
reported gradient-based interpolation algorithm which only indi- micropolarizer for 90r linear
cates the image’s directional change along 0◦ , 90◦ , 45◦ and 135◦ , 90º Micro-polarizer
polarized component
the presented smoothness-based algorithm covers the variations
along all the possible directions, leading to more accurate se- micropolarizer for 45r linear
45º Micro-polarizer
polarized component
lection between the bilinear and bicubic algorithms. According
to our extensive simulation results, the proposed implementation micropolarizer for 135r linea
exhibits the lowest mean square error (MSE) for the test images 135º Micro-polarizer
polarized component
among all the previously reported algorithms, including bilinear,
bicubic and gradient-based interpolation algorithms.

I. I NTRODUCTION Fig. 1. Typical 2×2 micro-polarizer pattern for DoFP sensors.

Recent advancement in the nanofabrication field has en-

abled the promising concept of division of focal plane po-
larimeters (DoFP), which is also known as the polarization
imaging based on the focal plane micro-polarimeters with
predefined pattern [1]–[5]. Similar to the color filter array
(CFA) widely exploited in the color image sensor domain [6],
traditional monochromatic image sensors can be endowed with
the capability of polarization imaging by integrating a layer
of pixelated micro-polarizers on top [1]–[5]. Consequently,
as shown in Fig. 1, in most literatures, the micro-polarizers
are arranged to a 2×2 periodic pattern, with their four local
major polarization axes oriented typically along 0◦ , 90◦ , 45◦ ,
135◦ [1]–[3], [5]. In the CFA-based color image sensors, it
is well-known that: in order to have the different wavelength
channels (i.e. R, G and B) simultaneously extracted, the spatial
resolution of each channel needs to be compromised. As
a result, different interpolation algorithms are necessary to
compensate the spatial resolution loss, namely “demosaicing”
[6], [7]. Here for the DoFP-based polarization imaging, we
can capture different polarization channels in one single frame,
however, the polarization image sensors suffer from the similar
problem of spatial resolution loss and low accuracy of captured
polarization images, hence the same concept of “polarization
demosaicing” applies as well [8].
Since the image is sampled through four pixelated micro-
polarizers scattered throughout the space, the pixels in the
aforesaid 2×2 periodic pattern record different polarization
information of the instantaneous field of view. Therefore, if
we were to directly calculate the polarization related physi-
cal parameters according to those polarization signatures, an
error is introduced as there will be a deviation from the
true polarization information of the target [8]. Additionally,
the micro-polarizers on the imaging array are oriented in
four different directions and each records one of the four
polarization signatures. This results in down-sampled sub-
images and the reduction of spatial resolution. In order to
improve both the spatial resolution and the accuracy of the
polarization information, it is necessary to interpolate the
images obtained from DoFP polarization sensors. Currently,
several interpolation algorithms exist, including bilinear, bicu-
bic and gradient-based algorithms [9]–[13]. The bilinear algo-
rithm has the lowest computational complexity but generally
yields the worst results especially when dealing with images
having complex spatial information [9]. The bicubic algorithm
may produce better results as indicated in [9], but suffers
from the high computational complexity. In order to alleviate
this limitation of computation complexity, the gradient-based
interpolation algorithm is introduced in [10], [11]. In this
algorithm, the gradient is calculated for 0◦ , 90◦ , 45◦ and
135◦ directions, respectively. If the gradient is greater than
a threshold value, the high-order interpolation algorithm (i.e.
bicubic interpolation algorithm) is used. Otherwise the bilinear
interpolation algorithm is adopted. However, the gradient is an
indicator of the directional change. It is straightforward that
the directional change can occur in any direction. Thus the
algorithm proposed in [10], [11], which only calculates the
gradient in four directions, may not produce the most accurate
results.
In this paper, a novel algorithm that aims to improve
the gradient-based interpolation algorithm is proposed. Instead
of using the gradient to represent the image variations, the
smoothness is exploited. If the smoothness variation of the
 f(x,y) f(x+1,y) f(iͲ1,jͲ1) f(i,jͲ1) f(i+1,jͲ1) f(i+2,jͲ1)

f(u,v) f(iͲ1,j) f(i,j) f(i+1,j) f(i+2,j)

f(i+u,j+v)

f(x,y+1) f(x+1,y+1)
f(iͲ1,j+1) f(i,j+1) f(i+1,j+1) f(i+2,j+1)

Fig. 2. The traditional bilinear interpolation algorithm.

f(iͲ1,j+2) f(i,j+2) f(i+1,j+2) f(i+2,j+2)

image is larger than a threshold value, the high-order interpo-

lation algorithm (i.e. bicubic) is used; otherwise, the bilinear Fig. 3. The traditional bicubic interpolation algorithm.
algorithm is adopted. The rest of this paper is structured as
follows: Section II covers a review of the bilinear and bicubic
interpolation algorithms; the proposed smoothness-based in-
terpolation algorithm is presented in Section III; Section IV The formula of bicubic interpolation is expressed as follows:
shows the results and compares the proposed interpolation f (i + u, j + v) = A · B · C (3)
algorithm to the previous ones; finally, the paper is concluded
in Section V. where A, B and C are matrices expressed as follows:
T
II. R EVIEW OF THE TRADITIONAL INTERPOLATION A = [ s(1 + u) s(u) s(1 − u) s(2 − u) ] (4)
ALGORITHMS FOR D O FP
2 3
f (i − 1, j − 2) f (i, j − 2) f (i + 1, j − 2) f (i + 2, j − 2)
A. Bilinear Interpolation Algorithm 6 f (i − 1, j − 1)) f (i, j − 1) f (i + 1, j − 1) f (i + 2, j − 1) 7
B=4 f (i − 1, j) f (i, j) f (i + 1, j) f (i + 2, j) 5
The principle of the bilinear interpolation algorithm ex- f (i − 1, j + 1) f (i, j + 1) f (i + 1, j + 1) f (i + 2, j + 1)
tends the one-line interpolation method to two-variable in
a two-dimensional grid. Specifically, the pixel value to be
T
interpolated is calculated by its four neighboring pixels with C = [ s(1 + v) s(v) s(1 − v) s(2 − v) ] (5)
different weights. From Fig. 2, to determine the pixel value
of the interpolated point f (u, v), the four neighboring pixel
points f (x, y), f (x + 1, y), f (x, y + 1), and f (x + 1, y + 1) III. P ROPOSED S MOOTHNESS -BASED I NTERPOLATION
are adopted as follows: A LGORITHM
f (u, v) = f (x, y) ∗ (x + 1 − u) ∗ (y + 1 − v) As above-mentioned, the gradient is an indicator of the
+f (x, y + 1) ∗ (x + 1 − u) ∗ (v − y) directional change, which can occur in any direction. The pre-
viously reported implementation only calculates the gradient
+f (x + 1, y + 1) ∗ (u − x) ∗ (v − y) in four directions [10], [11], which can be further improved
+f (x + 1, y) ∗ (u − x) ∗ (y + 1 − v) (1) to cover the directional change in other orientations. In this
section, in order to further elevate the interpolation accuracy of
It is known that the computational complexity is low for the the gradient-based algorithm [10], [11], we proposed to utilize
bilinear interpolation, however, the interpolation accuracy may the smoothness for selecting the more appropriate algorithm.
not be excellent for different image conditions. The fundamental idea behind the proposed method is: if the
smooth region of the image is encountered, the bilinear method
B. Bicubic Interpolation Algorithm is used; otherwise, the bicubic method is adopted. So the
key factor of the proposed method is how to compute the
Bicubic interpolation, also known as the cubic convolution smoothness of the image. In this paper, the pixels’ variance of
interpolation, considers 16 neighboring pixels (4×4) (Fig. 3), an image’s selected region is used to represent the smoothness.
which is different from the bilinear interpolation only taking The proposed method is described using a flow chart in Fig.
4 pixels (2×2) into account. For the image pattern with 4, which mainly involves the following steps:
large image variations, this interpolation algorithm generally
yields superior results, however, it is at the expense of high 1) Divide the given polarization image into small blocks,
computational complexity. This can be reflected by the base here the block size is set to be 2×2.
function of the bicubic interpolation: 2) Evaluate the smoothness of image blocks by com-
⎧ ⎫ puting their variances, respectively. The 2×2 block’s
⎪
⎪ 1 − 2|W |2 + |W |3 |W | < 1 ⎪ ⎪ variance σ is derived as follows:
⎪
⎨ ⎪
⎬
s(W ) = 2 3
4 − 8|W | + 5|W | − |W | 1  |W | < 2 1
⎪ ⎪
(2) J = [(f (x, y) − E)2 + (f (x + 1, y) − E)2
⎪
⎪ ⎪
⎪ 3
⎩ ⎭ +(f (x, y + 1) − E)2 + (f (x + 1, y + 1) − E)2 ] (6)
0 |W |  2

Fig. 4. Flow chart of the proposed smoothness-based interpolation algorithm.
√ method generates the results with the largest MSE for all four
σ= J (7)
where E is mean value of the block, f (x, y), f (x +
1, y), f (x, y + 1), f (x + 1, y + 1) are four provided
pixel values.
3) Compute the threshold value σmedian . Here the
threshold value is the median value of all the blocks’
variances.
4) Compare each block’s variance σ to the threshold
value σmedian . If σ < σmedian , it indicates the block
can be considered as a smooth region, and the bilinear
interpolation method is applied to this block; while
if σ > σmedian , the block is considered as a non-
smooth region, and the bicubic interpolation method
is applied.
It is known that 1 multiplication and 3 additions are re-
quired for calculating E, and 6 multiplications plus 7 additions
are required for calculating σ. By adding 1 multiplication and
3 additions for the bilinear interpolation of a 2 × 2 window,
we have altogether 8 multiplications and 13 additions. Com-
pared with the bicubic method requiring 70 multiplications
and 45 additions, the proposed smoothness-based interpolation
method can be executed with much lower computational
complexity. Regarding the gradient-based method [11], it is
difficult to have a fair comparison in term of computation
complexity. The reason is that the gradient-based method and
the smoothness-based method are two different criterias to sep-
arate pixels according to the interpolation algorithm adopted
(i.e. bilinear or bicubic), which can yield very different pixel
separation results with the same image, and is quite image-
dependent. Therefore, we choose to compare the accuracy of
the interpolation algorithm in the following section.
IV. R ESULTS AND D ISCUSSIONS
In order to effectively evaluate the performance of different
interpolation algorithms, high resolution polarization images
were obtained by rotating a linear polarization filter in front of
an 8-bit grey-scale CCD camera. These original full-resolution
images, which are not available even with the real DoFP
sensors, are adopted here as the ground truth for evaluating
different interpolation algorithms. Then they are down-sampled
and interlaced to generate the test images, which can well-
mimic the polarization images generated by the real DoFP
sensors. Successively, the four interpolation algorithms (i.e. bi-
linear, bicubic, gradient-based, smoothness-based) are applied
to these generated test images to compute the full-resolution
images. After that, these interpolated images are compared
with the ground-truth images in term of the mean square error
(MSE), which is commonly-used to evaluate the performance
of different interpolation algorithms [9]–[11]. Here the MSE
is defined by:
1
M SE = [Oc (i, j) − Ic (i, j)]2 (8)
MN
1iM 1jN
where M ×N is the test image’s array size, Oc (i, j) is the pixel
value of the ground truth polarization images, and Ic (i, j) is
the interpolated polarization images.
Generally speaking, lower MSE corresponds to higher
accuracy of the applied interpolation algorithm. As a result,
as shown in Table I, the proposed smoothness-based algorithm
is compared with the previously reported bilinear, bicubic and
the gradient-based algorithms, in term of MSE. It is observed
that among the four algorithms, the bilinear interpolation
polarization sub-images; while the proposed smoothness-based
method generates the results with the minimum MSE for all
four polarization sub-images. Meanwhile, it is noted that even
though the bicubic interpolation method has higher interpola-
tion order (i.e. computation compexity), it does not necessarily
leads to lower MSE or higher interpolation accuracy. In other
words, the interpolation accuracy is not closely related to the
interpolation order. For example, we have a test image in an ex-
tremely smooth case where all the pixels are equal to a constant
value. As a result, the best interpolation method should be the
low order interpolation method (e.g. zero-hold interpolation),
not the high-order bicubic interpolation. Therefore, if an image
is smooth enough, the bilinear method may lead to lower
MSE than the bicubic method. The proposed smoothness-
based interpolation method exhibits the highest accuracy since
it takes the smoothness of an image’s different blocks into
account. If the block of an image is relatively smooth, which
mainly corresponds to the low frequency components, the
low order interpolation method (i.e. bilinear) is preferred in
our case; otherwise, the bicubic method is adopted. Fig. 5
presents one set of the test images interpolated by four different
algorithms.
 TABLE I. MSE COMPARISON WITH THE PREVIOUS INTERPOLATION ALGORITHMS .

Bilinear Bicubic Gradient-based Smoothness-based

MSE of 0◦ 8.7423 5.7530 6.7545 5.6796
MSE of 90◦ 9.8282 6.5321 7.6749 6.4585
MSE of 45◦ 9.1837 6.1072 7.1235 6.0389
MSE of 135◦ 9.3820 6.1806 7.3626 6.1039

Bilinear Bicubic GradientͲBased SmoothnessͲBased

0e

(a) (b) (c) (d)

45e

(e) (f) (g) (h)

90e

(i) (j) (k) (l)

135e

(m) (n) (o) (p)

Fig. 5. One set of the test images interpolated by the four different algorithms.

V. C ONCLUSIONS [3] M. Kulkarni and V. Gruev, “Integrated spectral-polarization imaging

In this paper, a smoothness-based algorithm is proposed to
further improve the interpolation accuracy of the DoFP sen-
sors. Compared with all the previously demonstrated interpo-
lation algorithms for polarization images, the proposed imple-
mentation exhibits the highest accuracy with lowest MSE for
all different polarization channels. Moreover, the algorithm’s
overall computation complexity is well-balanced between the
traditional mainstream bilinear and bicubic algorithms. This
makes it quite suitable for the on-chip hardware implemen-
tation, which enables the promising concept of polarization
image sensing and processing together on a single chip.
ACKNOWLEDGMENT
This work was supported by the National Natural Sci-
ence Foundation of China (Grant No. 61601168, 61504087),
ICT Fund of UAE, the Fundamental Research Founda-
tion of Shenzhen (Grant No. JCYJ20150324141711677,
JCYJ20160520170741660, JCYJ20160308094919279) and the
Natural Science Foundation of SZU (Grant No. 2016054).
R EFERENCES
[1] V. Gruev and R. Perkins, “A 1 MPixel CCD Image Sensor with
Aluminum Nanowire Polarization Filter,” in Proc. IEEE International
Symposium on Circuits and Systems (ISCAS), pp. 629–632, 2010.
[2] V. Gruev, R. Perkins and T. York, “CCD polarization imaging sensor
with aluminum nanowire optical filters,” Opt. Express, vol. 18, no. 18,
pp. 19087–19094, 2010.
sensor with aluminum nanowire polarization filters,” Opt. Express, vol.
20, no. 21, pp. 22997–23012, 2012.
[4] K. Sasagawa, S. Shishido, K. Ando, H. Matsuoka, T. Noda, T. Tokuda,
K. Kakiuchi and J. Ohta, “Image sensor pixel with on-chip high extinc-
tion ratio polarizer based on 65-nm standard CMOS technology,” Opt.
Express, vol. 21, no. 9, pp. 11132–11140, 2013.
[5] T. York, R. Marinov and V. Gruev, “260 frames-per-second 648x488
resolution division-of-focal-plane polarimeter with structural dynamics
and tracking applications,” Opt. Express, vol. 24, no. 8, pp. 8243–8252,
2016.
[6] B. E. Bayer, “Color imaging array,” US Patent 3,971,065, 1976.
[7] O. Losson, L. Macaire, Y. Yang, “Comparison of Color Demosaicing
Methods,” Advances in Imaging and Electron Physics, vol. 162, pp. 173–
265, 2010.
[8] B. M. Ratliff, C. F. LaCasse and J. Scott Tyo, “Interpolation strategies for
reducing IFOV artifacts in microgrid polarimeter imagery,” Opt. Express,
vol. 17, no. 11, pp. 9112–9125, 2009.
[9] S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for
division of focal plane polarimeters,” Opt. Express, vol. 19, no. 27, pp.
26161–26173, 2011.
[10] S. Gao and V. Gruev, “Gradient Based Interpolation for Division of
Focal Plane Polarization Imaging Sensors,” in Proc. IEEE International
Symposium on Circuits and Systems (ISCAS), pp. 1855–1858, 2012.
[11] S. Gao and V. Gruev, “Gradient-based interpolation method for division-
of-focal-plane polarimeters,” Opt. Express, vol. 21, no. 1, pp. 1137–1151,
2013.
[12] J. Zhang, H. Luo, B. Hui and Z. Chang, “Image interpolation for
division of focal plane polarimeters with intensity correlation,” Opt.
Express, vol. 24, no. 18, pp. 20799–20807, 2016.
[13] E. Gilboa, J. P. Cunningham, A. Nehorai and V. Gruev, “Image
interpolation and denoising for division of focal plane sensors using
Gaussian processes,” Opt. Express, vol. 22, no. 12, pp. 15277–15291,
2014.