A Correlation-Based Interpolation Algorithm for

Division-of-Focal-Plane Polarization Sensors

Xiaoxiao Xu∗, Meenal Kulkarni∗, Arye Nehorai and Viktor Gruev
Washington University in St Louis, 1 Brookings Drive, St Louis, United States

ABSTRACT
We propose an interpolation algorithm for Division-of-Focal-Plane (DoFP) polarimeters based on the correlation
between neighboring pixels. DoFP polarimeters monolithically integrate pixelated nanowire polarization filters
with an array of imaging elements. DoFP sensors have been realized in the visible and near-infrared regime. The
advantages of DoFP sensors are twofold. First, they capture polarization information at every frame. Second,
they are compact and robust. The main disadvantage is the loss of spatial resolution due to the super-pixel
sampling paradigm at the focal plane. These sensors produce four low-resolution images, where each image
has been recorded by a linear polarization filter offset by 45 degrees. Our algorithm addresses the loss of
spatial resolution by utilizing the correlation information between the four polarization pixels in a super-pixel
configuration. The method is based on the following premise: if one or more of three polarization parameters
(angle of polarization, degree of polarization, and intensity) are known for a spatial neighborhood, then the
unknown pixel values for the 0◦ image, for example, can be computed from the intensity values from the 45◦ ,
90◦ and 135◦ images. The proposed algorithm is applied to select cases and found to outperform the bicubic
spline interpolation method.
Keywords: Image Interpolation, Polarization, Division-of-Focal-Plane, Correlation-Based Interpolation

1. INTRODUCTION
The primary physical parameters of light are intensity, wavelength, and polarization. Human vision can only
perceive the first two parameters and encodes them as brightness and color respectively. However, polarization
can provide us extra information that is not correlated to the intensity and wavelength data.1 Thus it has
various applications. For example, polarization information richly describes material surface features,2 shapes3
and roughness,4 and can be used in material classification5 and diffusion-reflection component analysis.6 In
biological and clinical research, polarization cues assist in depth image acquisition and segmentation,7 and show
the status of scattered light reflected from animals,8 organs, etc, which offers a better understanding of animal
behavior and pathogenesis. Polarization techniques can even boost advances in computer vision and robotics.9
Generally, a light wave is neither perfectly polarized nor completely unpolarized. This is termed as partial
polarization. It is sufficient to represent the complete polarimetric properties of partially polarized light using
the Stokes parameter S0 (the total intensity of the light incident on the image sensor), the angle of polarization
ϕ, and degree of linear polarization ρ.1 This representation is widely applied.
A variety of polarization imaging sensors has been developed to record the polarization parameters of light.10
Among these imaging sensors, a particular class called Division-of-Focal-Plane polarimeters (DoFP),11, 12 also
known as microgrid polarimeters, has significant advantages. The DoFP systems include imaging elements and
nanowire polarization filters on the same substrate. Thus, all polarization measurements are simultaneously
obtained within a single image. As demonstrated in Fig. 1(a), four linear micropolarization filters offset by 45◦
are aligned in each 2 × 2 pixel region of a pixelated filter array. This region, which filters the incident light wave
and records the filtered light intensity with four different photodiodes, is called a super-pixel.
DoFP sensors capture polarization information in real time with high spatial resolution. Furthermore, they
are compact and robust and have no moving parts, unlike typical polarization-sensing systems which have
complicated optical setups. However, there exists one distinct limitation. The DoFP sensor samples the scene
through the 0◦ , 45◦ , 90◦ and 135◦ polarizers, and records the four sub-sampled images. Therefore, each pixel
∗Equally contributing authors

Polarization: Measurement, Analysis, and Remote Sensing X

edited by David B. Chenault, Dennis H. Goldstein, Proc. of SPIE Vol. 8364, 83640L
© 2012 SPIE · CCC code: 0277-786X/12/$18 · doi: 10.1117/12.919196

Proc. of SPIE Vol. 8364 83640L-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 Incident CTt)°I (Zt) (1)Cj
Light I

J
(ZZ)I

I
4-Polarize
FiIterArra,

(tI

Imaging Element

(a) (b)

Figure 1: Schematic figure of (a) Division-of-Focal-Plane polarization imaging sensor with a 4-polarizer filter
array, (b) I0◦ array as sampled by 0◦ filters.

performs only one out of the four necessary intensity measurements (Fig. 1(b)), leading to a loss of spatial
resolution. In order to obtain the polarization parameters at each pixel to reconstruct the original image, one
relies on the pixel’s neighbors to estimate the other three intensities. The estimation process will inevitably
introduce errors, as each pixel has a different instantaneous field of view.13 This problem is similar to the one
encountered for color images obtained from Bayer color filter array sensors (demosaicing).14
Numerous interpolation strategies have been investigated to enhance the accuracy of the reconstructed high
resolution image. Because of a similar sub-sampling technique to color images, some interpolation methods
have been directly imported from the color domain, for instance, the bilinear method13 and the bicubic spline
method.15 However, the inherently different physical principles behind capturing polarization images and color
images motivate the development of an interpolation method that is uniquely applicable to, and efficient for,
DoFP polarization images.
In this paper we present an interpolation algorithm that relies on correlation information between neighboring
pixels in order to partially recover the lost spatial resolution. In section 2, we provide an overview of computing
polarimetric information from DoFP sensors. In section 3, we describe the correlation-based interpolation strat-
egy. In section 4, we show the results of interpolating synthetic polarization images using the proposed method,
and the comparison with the bicubic spline method.

2. OBTAINING POLARIZATION INFORMATION FROM DOFP SENSORS

The intensity measurements from the DoFP sensor super-pixel are necessary and sufficient to compute the
polarization parameters as follows
1
S0 = (I(0◦ ) + I(45◦ ) + I(90◦ ) + I(135◦ )), (1)
2
S1 = I(0◦ ) − I(90◦ ), (2)
◦ ◦
S2 = I(45 ) − I(135 ), (3)
ϕ = (1/2) tan−1 (S2 /S1 ), (4)
√
ρ = S12 + S22 /S0 , (5)

where I(ψ), ψ = 0◦ , 45◦ , 90◦ , 135◦ is the intensity of the light wave recorded with linear polarization filter
oriented at θ angles. The parameters S0 , S1 and S2 are known as the first three Stokes parameters and provide
information about the linear polarization state of the incident light.

Proc. of SPIE Vol. 8364 83640L-2

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 The intensity response of a pixel coupled with a linear polarization filter at a given orientation can be
computed via the following equations
S0
I(0◦ ) = (1 + ρ · cos(2 × 0◦ − 2ϕ)), (6)
2
S0
I(45◦ ) = (1 + ρ · cos(2 × 45◦ − 2ϕ)), (7)
2
S0
I(90◦ ) = (1 + ρ · cos(2 × 90◦ − 2ϕ)), (8)
2
S0
I(135◦ ) = (1 + ρ · cos(2 × 135◦ − 2ϕ)). (9)
2

In general terms,
S0
Iψ = (1 + ρ · cos(2ψ − 2ϕ)). (10)
2
Eq. (10) was the original equation derived by Sir George Stokes in an effort to mathematically describe
unpolarized light. This equation will also form the basis of our interpolation method which is described in the
next section.

3. INTERPOLATION STRATEGY
Our strategy is inspired by the interpolation method which takes advantage of the correlation between red (R),
green (G) and blue (B) pixels in color images. Since there are spectral overlaps among the R, G, B channels, the
R and B values, for example, are somewhat correlated to the G value over the interpolation pixel neighborhood.
Then, the estimation of the missing values for one color can be expressed as a combination (for example, weighted
sum14 ) of the R, G, B colors. However, the overlapping spectral regions of R, G, B are not distinct, which makes
it difficult to define the correlation coefficients for the R, G, B values. Consequently, the interpolation model is
not entirely accurate, and complicated to build.
Contrary to color interpolation, the relationship between the I(0◦ ), I(45◦ ), I(90◦ ) and I(135◦ ) polarization
intensities is explicit in Eqs. (6)-(9). We exploit these relationships in order to reconstruct the I(0◦ ), I(45◦ ),
I(90◦ ) and I(135◦ ) arrays.
It should be noted that the proposed algorithm assumes that one or more of the three polarization parameters
ϕ, ρ and S0 has been previously estimated, i.e. some initial conditions are known. This estimation may be
obtained by knowing the nature of the imaged target. Furthermore, the algorithm presented in this paper can be
an integral part of an iterative interpolation process, where initial estimates of the polarization information are
computed using regular interpolation techniques. For instance, having carried out bicubic spline interpolation on
a DoFP image, we can identify regions of pixels that demonstrate one or more constant polarization parameters.
This can be the basis for a more precise interpolation using the proposed algorithm.
Clearly, there are eight possible combinations of known and unknown polarization parameters ϕ, ρ and S0 .
In the following subsections, we assume that two of the three parameters ϕ, ρ and S0 are known to us by some
means (as described above) and constant, with the third parameter varying. Therefore, we consider three cases.

3.1 Case 1: varying ρ, known ϕ, known S0

We consider the condition when the total incident intensity S0 and the angle of polarization ϕ are known to us.
The degree of linear polarization ρ is varying in an unknown manner. Thus our goal is to recover the unknown
ρ and then the unknown intensities of ψ angle filtered light waves.
We employ pixel (1,1) to demonstrate our method. From Eq. (10), the degree of linear polarization ρ(1, 1)
at a pixel (1,1) of array Iψ (see Fig. 1(b)), is given as

2Iψ (1, 1) − 1
ρ(1, 1) = . (11)
cos(2ψ − 2ϕ(1, 1))

Proc. of SPIE Vol. 8364 83640L-3

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 As I0◦ (1, 1) and ϕ(1, 1) are known at pixel (1,1), ρ(1, 1) may be directly computed by letting ψ = 0◦ . After
obtaining ρ(1, 1), we can calculate the unknown intensity values for the other three (Iψ , ψ = 45◦ , 90◦ , 135◦ )
polarization arrays at pixel (1,1), using Eq. (10)
S0 (1, 1)
Iψ (1, 1) = (1 + ρ(1, 1) · cos(2ψ − 2ϕ(1, 1))). (12)
2
3.2 Case 2: varying ϕ, known ρ, known S0
Again, we consider the array Iψ . We now explore the condition when the degree of linear polarization ρ(1, 1)
and total incident intensity S0 (1, 1) are known. We may then calculate the angle of polarization ϕ(1, 1) using
Eq. (10) and ψ = 0◦ [ ]
1 2Iψ (1, 1)/S0 (1, 1) − 1
ϕ(1, 1) = ψ − cos−1 . (13)
2 ρ(1, 1)
Having found the angle of polarization ϕ(1, 1), we can now calculate the unknown polarization intensity values
at pixel (1,1) using Eq. (12).

3.3 Case 3: varying S0 , known ϕ, known ρ

In the final case, we examine the condition when the total incident intensity is varying, and the angle ϕ and
degree ρ of polarization are known. We establish S0 as
2Iψ (1, 1)
S0 (1, 1) = . (14)
1 + (ρ(1, 1) · cos(2ψ − 2ϕ))

Having established the incident intensity S0 (1, 1), we can now calculate the unknown polarization intensity
values at pixel (1,1) using Eq. (12), as in Sections 3.1 and 3.2.

4. PERFORMANCE EVALUATION
We employed artificial images to numerically evaluate the performance of our interpolation strategy. The per-
formance of the proposed method is then compared with that of a bicubic spline interpolation, which has been
established as the most effective interpolation method for DoFP polarization images through the evaluation of a
suite of algorithms.15
As the intensity S0 , the angle of polarization ϕ, and the degree of linear polarization ρ comprise all polarization
information, we first generated an M × N array that encompasses these parameters. We simulated this scene at
the focal plane of the detector by modulating these polarization properties on spatial sinusoids of the form
t(x, y) = cos(2πfx x + 2πfy y), (15)
where x and y indicate the pixel position, and fx and fy are spatial frequencies in cycles per detector pixel.13
The full resolution images were then decimated to form the four low resolution arrays. After the decimation
procedure, we applied each interpolation method to the subsampled images to reconstruct the original images.
We used the measures described in the following to analyze the performance of the two interpolation methods.

4.1 RMSE comparison

The root mean squared error (RMSE) is computed as
√
1 ∑ ∑
RMSE = (T (i, j) − I(i, j))2 , (16)
MN
1≤i≤M 1≤j≤N

where T (i, j) denotes the true intensity and I(i, j) is the interpolated intensity at pixel (i, j). Clearly, the lower
the RMSE, the better the interpolation method.
A comparison of RMSEs for the I(0◦ ) array over spatial frequency for the different interpolation methods is
shown in Fig. 2. It is seen that the RMSE of the bicubic interpolation method is low upto fx and fy around 0.32,
rapidly increasing past this point. However, it can be seen that the correlation-based algorithm has negligible
RMSE inspite of increase in spatial variations across the image.

Proc. of SPIE Vol. 8364 83640L-4

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 −17
x 10
0.2
RMSE I0

RMSE I0
0.1 0.5 0.5
3
0
0.4 0.4
0 2
0.1 0.3 0 0.3
0.1
0.2 0.2
0.2 0.2
0.3
0.3 0.1
0.1
0.4 fy 0.4 fy
fx 0.5 0 0.5 0
fx

(a) (b)

−9
x 10
0.4 1
RMSE I0

RMSE I0

0.2 0.5 0.5

0.5
0
0.4 0.4
0 0
0.1 0.3 0 0.3
0.2 0.1
0.2 0.2
0.2
0.3
0.1 0.3 0.1
0.4 fy 0.4 fy
fx 0.5 0 0.5 0
fx

(c) (d)

−17
x 10
RMSE I0

0.5
4
RMSE I0

0.5
0.5
0 2
0 0 0.4
0.4
0.1 0
0.3 0.3
0.1
0.2
0.2 0.2 0.2
0.3
0.3
0.1 0.1
0.4 0.4 fy
fy
fx 0.5 0 0.5 0
fx

(e) (f)

Figure 2: A comparison of RMSE of I(0◦ ) array for (a) Bicubic spline interpolation and (b) Correlation-based
algorithm presented in section 3.1, (c) Bicubic spline interpolation and (e) Correlation-based algorithm presented
in section 3.2, and (e) Bicubic spline interpolation and (f) Correlation-based algorithm presented in section 3.3.

Proc. of SPIE Vol. 8364 83640L-5

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 1.2 1.2

1 1

0.8 0.8
MTF

MTF
0.6 0.6

0.4 0.4

0.2 0.2

0 0

0.2 0.2

0.4 0.4
0.4 0.5 0.4 0.5
0.2 0.3 0.2 0.3
0.6 0 0.1 0.6 0 0.1
fx fy fx fy

(a) (b)

Figure 3: A comparison of MTF for (a) Bicubic spline interpolation, and (b) Correlation-based algorithm. fx
and fy are the normalized spatial frequencies in x and y direction.

4.2 MTF Comparison

Another means of comparison is the modulation transfer function (MTF)13 of intensity S0 over spatial frequency.
The image sensor is a spatial sampling device; thus the sampling theorem applies here. The sampling theorem
sets the limits for the reproducibility in space of the input spatial frequencies; that is, the spatial frequency
components higher than the respective Nyquist rate (fN ) cannot be faithfully reproduced and are instead aliased.
MTF is the spatial frequency response of an imaging sensor; it provides a measure of how well the scene can
be reconstructed depending on its spatial frequency. Specifically, a high MTF at the higher spatial frequencies
indicates that the interpolation method is successfully able to reconstruct high spatial frequency information.
To compute the MTF of S0 for each set of (fx , fy ), we represent the S0 of both the full resolution true image
(TS0 ) and the interpolated image (IS0 ), in the frequency domain.

F TS0 = FFT{TS0 }, F IS0 = FFT{IS0 }, (17)

where FFT denotes the Fast Fourier Transform.

The modulation terms of the spatial sinusoids are delta functions; so we know their positions are at (fx +
M/2, fy + N/2). Then, the MTF for (fx , fy ) is

F IS0 (fx + M/2, fy + N/2)

MTF(fx , fy ) = . (18)
F TS0 (fx + M/2, fy + N/2)

Sweeping fx and fy in [0, 0.5], we obtained the MTF against different spatial frequencies. The comparison
of the MTF over spatial frequency for the two interpolation methods is shown in Fig. 3. A slice is taken from
this plot at fx = fy and presented in Fig. 4.
It is seen that bicubic spline interpolation fails to satisfactorily reproduce spatial frequency components past
a spatial frequency of about 0.32. In contrast, the proposed correlation-based interpolation algorithm is able
to successfully reconstruct spatial frequency components right up to the Nyquist point. This implies that even
when there is rapid variation across the image, the interpolation method will be effective.

Proc. of SPIE Vol. 8364 83640L-6

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 1.4
Correlation
Bicubic interpolation
1.2

0.8

MTF
0.6

0.4

0.2

0
0.1 0.2 0.3 0.4 0.5 0.6
Spatial frequency (fx = fy)

Figure 4: A comparison of MTF of bicubic spline interpolation and the proposed algorithm at fx = fy .

5. CONCLUSION
An interpolation method has been proposed, which exploits the inherent correlation between the four pixels of a
DoFP sensor to reconstruct the individual 0◦ , 45◦ , 90◦ and 135◦ intensity arrays, and the polarization parameters.
The method assumes that some of the polarization parameters have been estimated previously and is intended to
be a later step in an iterative estimation process. Three cases were considered, in each of which two parameters
are assumed known. The performance of this correlation-based interpolation algorithm was evaluated using two
measures, RMSE and MTF. It was compared in terms of these two metrics to the state-of-the-art bicubic spline
interpolation method and found to remarkably exceed its reconstruction ability for the cases considered.

ACKNOWLEDGMENTS
Author V.G. would like to thank the support of this work by National Science Foundation (NSF) grant number
1130897, and Air Force Office of Scientific Research grant number FA9550-10-1-0121. Author A.N. would like
to thank the support of this work by National Science Foundation (NSF) grant number CCF-0963742, and Air
Force Office of Scientific Research grant number FA9550-11-1-0210.

REFERENCES
[1] Collett, E., [Polarized light: Fundamentals and applications], Optical Engineering, New York (1992).
[2] Morel, O., Stolz, C., Meriaudeau, F., and Gorria, P., “Active lighting applied to three-dimensional recon-
struction of specular metallic surfaces by polarization imaging,” Appl. Opt. 45(17), 4062–4068 (2006).
[3] Wolff, L., “Surface orientation from polarization images,” in [Proc. SPIE], 850, 110 (1987).
[4] Azzam, R. and Bashara, N., “Polarization characteristics of scattered radiation from a diffraction grating
by ellipsometry with application to surface roughness,” Phys. Rev. B 5(12), 4721 (1972).
[5] Wolff, L., “Polarization-based material classification from specular reflection,” IEEE Trans. Pattern Anal.
Mach. Intell. 12(11), 1059 –1071 (1990).
[6] Umeyama, S. and Godin, G., “Separation of diffuse and specular components of surface reflection by use
of polarization and statistical analysis of images,” IEEE Trans. Pattern Anal. Mach. Intell. 26(5), 639–647
(2004).
[7] Wallace, A., Liang, B., Trucco, E., and Clark, J., “Improving depth image acquisition using polarized light,”
Int. J. Comput. Vision 32(2), 87–109 (1999).

Proc. of SPIE Vol. 8364 83640L-7

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms

 [8] Nilsson, D. and Warrant, E., “Visual discrimination: Seeing the third quality of light,” Curr. Biol. 9(14),
535–537 (1999).
[9] Wolff, L., “Polarization camera for computer vision with a beam splitter,” JOSA A 11(11), 2935–2945
(1994).
[10] Tyo, J. S., Goldstein, D. L., Chenault, D. B., and Shaw, J. A., “Review of passive imaging polarimetry for
remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006).
[11] Perkins, R. and Gruev, V., “Signal-to-noise analysis of stokes parameters in division of focal plane polarime-
ters,” Opt. Express 18(25), 25815–25824 (2010).
[12] York, T. and Gruev, V., “Optical characterization of a polarization imager,” in [Proc. IEEE Int. Symp.
Circuits and Systems], 1576–1579 (2011).
[13] Ratliff, B., LaCasse, C., and Tyo, J., “Interpolation strategies for reducing ifov artifacts in microgrid
polarimeter imagery,” Opt. Express 17(11), 9112–9125 (2009).
[14] Malvar, H., He, L., and Cutler, R., “High-quality linear interpolation for demosaicing of bayer-patterned
color images,” in [Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing], 3, 485–488 (2004).
[15] Gao, S. and Gruev, V., “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,”
Opt. Express 19(27), 26161–26173 (2011).

Proc. of SPIE Vol. 8364 83640L-8

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/22/2015 Terms of Use: http://spiedl.org/terms