Letter Vol. 43, No.

14 / 15 July 2018 / Optics Letters 3265

Sparse representation-based demosaicing

method for microgrid polarimeter imagery
JUNCHAO ZHANG,1,2,3,4,5,* HAIBO LUO,1,4,5 RONGGUANG LIANG,3 ASHFAQ AHMED,6 XIANGYUE ZHANG,1,2,4,5
BIN HUI,1,4,5 AND ZHENG CHANG1,4,5
1
Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China
2
University of Chinese Academy of Sciences, Beijing 100049, China
3
College of Optical Sciences, University of Arizona, Tucson, Arizona 85721, USA
4
Key Laboratory of Opto-Electronic Information Processing, CAS, Shenyang 110016, China
5
The Key Lab of Image Understanding and Computer Vision, Liaoning Province, Shenyang 110016, China
6
Department of Bioengineering, Hong Kong University of Science and Technology, Hong Kong, China
*Corresponding author: zhangjunchao@sia.cn

Received 17 April 2018; revised 6 June 2018; accepted 12 June 2018; posted 13 June 2018 (Doc. ID 328604); published 6 July 2018

To address the key image interpolation issue in microgrid
polarimeters, we propose a machine learning model based
on sparse representation. The sparsity and non-local self-
similarity priors are used as regularization terms to enhance
the stability of an interpolation model. Moreover, to make the
best of the correlation among different polarization orienta-
tions, patches of different polarization channels are joined
to learn adaptive sub-dictionary. Synthetic and real images
are used to evaluate the interpolated performance. The exper-
imental results demonstrate that our proposed method
achieves state-of-the-art results in terms of quantitative mea-
sures and visual quality. © 2018 Optical Society of America
OCIS codes: (100.3020) Image reconstruction-restoration;
(100.6640) Superresolution; (110.5405) Polarimetric imaging;
(120.5410) Polarimetry; (260.5430) Polarization.
https://doi.org/10.1364/OL.43.003265
Polarimetric imaging provides useful information related to
surface roughness, shape, and normal direction and is invisible
on intensity or spectral images. Thus, it has been successfully
applied in many fields such as material classification [1], 3-D
surface reconstruction [2], dehazing [3], and biomedical imag-
ing [4]. Microgrid polarimeters, i.e., division-of-focal-plane
(DoFP) polarimeters, are composed of a collection of pixelated
polarization filters aligned upon a focal plane array.
However, polarization images captured by DoFP polarime-
ters are in mosaic type, as shown in Fig. 1. Each pixel records
only one out of four necessary intensity measurements, and
each individual pixel within a 2-by-2 super-pixel aligns to dif-
ferent polarization orientations. Spatial resolution is reduced,
and the reconstructed polarization property is not accurate
due to different instantaneous field of views (IFoVs) for each
pixel within a 2-by-2 super-pixel. To mitigate IFoV errors,
many interpolation methods [5–9] have been proposed for
DoFP images. However, all these methods are based on
polynomial approximation to estimate the missing pixel from
its local neighborhood, and they are unable to adapt to varying
structures in images. They generally tend to destroy the inher-
ent structures of Stokes parameters, which leads to the gener-
ation of artifacts in degree of linear polarization (DoLP, defined
in Ref. [7]) images and the reconstruction of inaccurate polari-
zation information.
In this Letter, we propose a new interpolation model based
on sparse representation. The proposed demosaicing model is
derived as follows.
For each polarization channel, assume that an observed
DoFP image y θ is directly down-sampled from its full-
resolution image xθ ; the formulation can be expressed as
y θ
Dθ x θ , (1)
where θ
f0°, 45°, 90°, 135°g and Dθ represent a down-
sampled matrix with the sampling factor of 2 along the hori-
zontal and vertical axes. To make the best of the correlation
among different polarization channels, images at four polariza-
tion channels are joined to build the interpolation model.
Thus, the sparse representation αi of joint patch R i X can
be expressed as
R i X
Φαi , (2)
where Φ is the dictionary, R i X
R i x 0° T , R i x 45° T ,
R i x 90° T , R i x135° T T , and R i is the matrix extracting a local
patch from X at position i. According to sparse representation
theory, the DoFP image interpolation question can be trans-
formed to minimize the following problem:

X XK 
min ky θ − Dθ x θ k22  γ kR i X − Φαi k22  λRα ,
α
θ i
1
(3)
where K is the number of partitioned patches, R· is the
regularization term, and γ and λ are the multiplier and
regularization parameter, respectively.

0146-9592/18/143265-04 Journal © 2018 Optical Society of America

 3266 Vol. 43, No. 14 / 15 July 2018 / Optics Letters
To solve the model formulated in Eq. (3), we first determine
the regularization terms. The sparsity prior is adopted for cod-
ing coefficient αi to constrain the representation of joint patch
R i X sparse. The sparsity of αi can be characterized by kαi k0
[10,11]. Non-local self-similarity refers to the fact that there
are often many repetitive patches throughout a natural image,
and such non-local redundancy is very useful to improve
the quality of reconstructed images. It has been successfully
adopted in image restoration [12,13]. Fortunately, for DoFP
images, non-local redundancy also exits in each channel.
Moreover, different channels have similar patterns. Therefore,
non-local self-similarity prior is used as an additional regulari-
zation term to improve the interpolated results.
As shown in Fig. 1, for each patch of size p × p in a DoFP
image, it can be expanded into a new patch of size p × p × 4
by filling up the missing pixels for each channel. Next, the
new patch is rearranged into a vector, denoted by x i

R i x0° T , R i x45° T , R i x90° T , R i x135° T T . Then we can
get its similar patches in a large enough local window. A patch
x ki is selected as a similar patch to xi if the Euclidean distance
between them is not greater than a preset threshold. Afterwards,
we get similar patches denoted by fx1i , x 2i , …, xni g. For a
given dictionary Φi, these coding coefficients of similar patches
can be average weighted Pto estimate the coding coefficient αi of
x i . We denote βi
nk
1 ωk αki , where αki
ΦTi xki and ωk
are calculated by solving the following regularized least square
problem:
ω̂
arg minfkxi − Sωk22  εkωk22 g,
ω
where S
x 1i , x2i , …, xni , ω
ω1 , ω2 , …, ωn T and ε is
the regularization parameter. The solution of Eq. (4) can be
obtained as
ω̂
ST S  εI−1 ST xi :
Since βi is the good estimation of αi , the difference
between them should be as small as possible. P Thus, we can
2
characterize the non-local self-similarity as i kαi − βi k2 .
The demosaicing model can be rewritten as

X
X
K
min ky θ − Dθ x θ k22  γ kR i X − Φαi k22
α
θ i
1
 to update the full-resolution image fx 0° , x45° , x 90° , x135° g. Such
X
K X
K
λ kαi k0  η kαi − βi k22 :
i
1 i
1
The re-weighted l2 norm showed better and more sparse re-
sults in Ref. [14]. Therefore, we modify the above model into a
re-weighted form as

X X
K 4p
min ky θ − Dθ x θ k22  γ kR i X − Φαi k22
α j
1
θ i
1
X
K X 4p
K X 2 
λ kαi k0  ηi,j kαi,j − βi,j k22
i
1 i
1 j
1
where αi,j and βi,j are the jth element of αi and βi , respectively.
A dictionary is very important to reconstruct a polarization
property. To keep the correlation among different channels,
corresponding patches at four polarization channels are joined
to learn adaptive sub-dictionary. For each joint patch xi , the
adaptive sub-dictionary can be obtained by solving the problem
Letter
minfjjxi − Φi αi k22  ζjjαi jj1 g. This is a joint optimization
problem, and it can be solved by alternatively optimizing Φi
and αi . However, solving the minimization problem directly
requires much computational cost. Thus, approximation solu-
tions can be used to address this problem. Since principal com-
ponent analysis (PCA) basis [13] has shown promising results
over other traditional bases for sparse representation, we use
PCA basis to approximate the solution. Recall that in non-local
self-similarity, for each joint patch xi , we get a set of non-local
similar patches, denoted by X i
x1i , x 2i , …, xm
i . Then the
adaptive sub-dictionary Φi for x i can be obtained by using
PCA technology [15,16]:
Φi
P T , (8)
where P is the eigenvector matrix calculated by performing
eigenvalue decomposition to the covariance matrix of X i .
If we set Φi
PT and αi
ΦTi x i , we will have
jjx i − Φi αi ΦTi
jjx i − Φi ΦTi x i k22
0, and the corre-
sponding sparsity regularization term will have a certain
amount.
In fact, all these sub-dictionaries form an over-complete
dictionary Φ
Φ1 , Φ2 , …, ΦK . For each joint patch xi ,
the corresponding adaptive sub-dictionary Φi is selected to
code it. This makes the coding coefficients of xi over other
sub-dictionaries zero, which leads to a very sparse representa-
tion of x i . That is, the adaptive sub-dictionary inherently
ensures the sparsity of the coding coefficients. Thus, over adap-
(4) tive sub-dictionaries, the demosaicing model can be modified
by removing the sparsity regularization term:

X X
K
min ky θ − Dθ x θ k22  γ kR i X − Φi αi k22
α
θ i
1
(5) 2 
X 4p
K X
 ηi,j kαi,j − βi,j k22 : (9)
i
1 j
1
Given a current estimation of image fx 0° , x45° , x 90° , x135° g,
the sparse representation fαi g can be updated for the next
iteration of minimization. After updating the PCA dictionary
Φ, the sparse representation and dictionary, in turn, are used
an iterative process terminates until the stopping rule is met.
(6)
For given images, the sparse representation in the sth iteration
can be obtained by minimizing the following function:
α̂i
arg min Jαi 
αi
X
2

γkR i X − Φi αi k22  ηi,j kαi,j − βi,j k22 : (10)
This is a quadratic problem, and we can get a closed-form
, (7) solution by setting ∂J∕∂αi
0:
ΦTi R i X s j  ηi,j βi,j ∕γ
αs1
i,j
: (11)
1  ηi,j ∕γ
After updating coding coefficients fαi g, we estimate full-
resolution image xs1
θ for each channel in the sth iteration
by solving the following minimization problem:
 Letter Vol. 43, No. 14 / 15 July 2018 / Optics Letters 3267

X
K
x̂ θ
arg min l xθ 
ky θ − Dθ xθ k22  γ kR i X − Φi αi k22 :
xθ
i
1
(12)
This is also a quadratic problem, and its closed-form solution
can be obtained by setting ∂l ∕∂xθ
0:
 XK −1 
s1 T T
xθ
Dθ Dθ  γ R i R i θ DTθ y θ
i
1

X
K 
γ R Ti Φi αi θ :
i
1
The overall demosaicing algorithm is summarized in
Algorithm 1. The computational complexity of the PCA sub-
dictionary is OK mp2  m2 p4  p6 . For sparse representa-
tion and full-resolution image updating, the complexities are
OK np2  p4  and ON 3  N 2 ∕4  K N p2  with N pixel
numbers of full-resolution image, respectively. The total com-
putational complexity is represented as OT N 3  N 2 ∕4 
K p6  m2  1p4  m  n  K N p2 .
Algorithm 1: Demosaicing Algorithm for DoFP Images
Input: DoFP image.
Initialization: Initialize iterator s
0, maximum iteration number T
and set:
(1) the initial high-resolution image xθ using the bicubic interpolation
method and PCA sub-dictionaries.
(2) the related parameters: η
1.2, γ
0.1, n
23, image block
size: 5 × 5.
Main Iteration: Increment s by 1 and perform the following steps:
1. Calculate sparse representation fαi g by Eq. (11).
2. Calculate image xs1
θ by Eq. (13).
3. Update the adaptive dictionaries Φ
Φ1 , Φ2 , …, ΦK  by Eq. (8).
4. Stopping rule: if s > T , stop. Otherwise, do another iteration.
Output: High-resolution image x θ and reconstructed Stokes
parameters and DoLP.
To provide a quantitative comparison of different interpo-
lation methods, we captured ground-truth images of four
polarization orientations for static scenes. This apparatus
includes a gray-scale camera (DMK 22AUC03) and a
Newport 10LP-VIS-B linear polarizer which is mounted on
a rotation stage (Thorlabs PRM1Z8E) and placed in front of
the camera. The rotation angles were distributed every 45° from
0° to 135°. Each image was sampled 300 times and averaged
to reduce the effects of random noise on interpolated results.
Next, we generated synthetic images as the sampling pattern
shown in Fig. 1. Then we evaluated the interpolated perfor-
mance by visual quality and quantitative measurement such
(13)
Fig. 2. Reconstructed DoLP images of different methods on syn-
thetic image building1. (a) Bicubic, (b) spline, (c) [7], (d) [8], (e) ours,
and (f ) ground-truth.
as peak signal-to-noise ratio (PSNR). Besides, real DoFP im-
ages were also used to verify the feasibility of our proposed
method. Real images were captured by our self-developed
DoFP camera in the visible waveband (752 × 582 pixels; ex-
tinction ratio, 9.58; transmittance, 71%). We compared our
proposed method with several mainstream methods such as bi-
cubic, spline [6], gradient-based [7], and our previous work [8].
As shown in Figs. 2 and 3, two scenes were used to evaluate
the reconstructed performance. Because of page limitation,
only the reconstructed DoLP images are shown. bicubic and
spline methods reconstruct noisy DoLP images because they
perform interpolation for each channel separately and intro-
duce serious interpolation noise in intensity images. Method
[7] adopts an image gradient prior, and the correlation among
different polarization channels is used in Ref. [8]. These two
methods produce better results for local structures such as
edges. However, all these methods are based on polynomial
approximation to estimate missing pixel from its local neigh-
borhood, and they are unable to adapt to varying structures
in images. Thus, the reconstructed DoLP images by these
methods are noisy. We can see that the reconstructed DoLP
images by using our proposed method are much cleaner while
preserving image details, and most closely resemble the ground-
truth, which is due to non-local self-similarity constraint
and adaptive sub-dictionary. Besides, the PSNR is used as a
quantitative evaluation criterion, and the convergent curves
of PSNR are shown in Fig. 4. One can see that our method

Fig. 3. Reconstructed DoLP images of different methods on

Fig. 1. Layout of a microgrid polarimetric sensor (left) and a synthetic image building2. (a) Bicubic (b) spline, (c) [7], (d) [8],
diagram of patch processing (right). (e) ours, and (f ) ground-truth.
 3268 Vol. 43, No. 14 / 15 July 2018 / Optics Letters Letter

Fig. 4. Convergent curves of the PSNR on synthetic images.

(a) building1 and (b) building2.

Table 1. PSNR Results of Different Methods on

Synthetic Images
Bicubic Spline [7] [8] Ours
Building1 x0 43.6204 43.6637 42.9119 43.3687 44.8913
x 45 43.9919 44.0241 43.2624 43.6699 45.8872
x 90 43.5886 43.6960 42.7822 43.2158 45.6223 Fig. 5. Reconstructed S 0 and DoLP images on a real DoFP image.
x 135 43.3862 43.4497 42.6463 43.1386 45.1869 From top to bottom: bicubic method, spline method, method [7],
S0 40.5980 40.7320 39.7920 40.2290 41.7803 method [8], and our method.
DoLP 40.2724 40.1530 40.0087 40.1801 42.1704
Building2 x 0 41.2516 41.5493 40.4113 40.7972 43.8248
x 45 41.9349 42.2285 41.0541 41.4445 45.0725
x 90 41.2407 41.4995 40.4677 40.7732 44.5211 method achieves state-of-the-art results in terms of quanti-
x 135 41.4361 41.6823 40.5736 40.9657 44.0594 tative measures and visual quality.
S0 38.3817 38.7292 37.5283 37.9144 40.5064
DoLP 37.2341 37.4525 36.5894 36.9499 40.3622 Funding. China Scholarship Council (CSC)
(201704910730); Thirteenth Five-Year Preresearch Program
of China (41415020104).
converges with the increasing iteration number. Based on the
computational complexity and experimental results, the itera- REFERENCES
tion number T is set to 50. The results on synthetic images are
listed in Table 1, where x θ is the intensity image at polarization 1. F. Hu, Y. Cheng, L. Gui, L. Wu, X. Zhang, X. Peng, and J. Su, Appl.
Opt. 55, 8690 (2016).
orientation θ. The bold number represents the best results. 2. F. Drouet, C. Stolz, O. Laligant, and O. Aubreton, Opt. Lett. 39, 2955
One can see that our proposed method achieves a much higher (2014).
PSNR than others, which demonstrates that our method recon- 3. S. Fang, X. Xia, X. Huo, and C. Chen, Opt. Express 22, 19523 (2014).
structs state-of-the-art results. 4. S. Alali and A. I. Vitkin, J. Biomed. Opt. 20, 061104 (2015).
A real car image was captured by a DoFP camera on a snowy 5. B. M. Ratliff, C. F. LaCasse, and J. S. Tyo, Opt. Express 17, 9112
day. The reconstructed DoLP images are shown in Fig. 5. The S 0 (2009).
6. S. Gao and V. Gruev, Opt. Express 19, 26161 (2011).
and DoLP images are reconstructed by the following methods: 7. S. Gao and V. Gruev, Opt. Express 21, 1137 (2013).
bicubic, spline, [7,8], and ours, from top to bottom. Strong 8. J. Zhang, H. Luo, B. Hui, and Z. Chang, Opt. Express 24, 20799
serrated artifacts occur at edges in the DoLP images reconstructed (2016).
by the bicubic and spline methods. Methods [7] and [8] produce 9. A. Ahmed, X. Zhao, V. Gruev, J. Zhang, and A. Bermak, Opt. Express
much better results than the above methods, and they mitigate 25, 10651 (2017).
serrated artifacts. However, method [7] brings slight edge distor- 10. M. Elad and M. Aharon, IEEE Trans. Image Process. 15, 3736
(2006).
tion at the top of the rear window. The last row shows the
11. M. Aharon, M. Elad, and A. Bruckstein, IEEE Trans. Signal Process.
reconstructed S 0 and DoLP images by using our method. We 54, 4311 (2006).
can see that our method produces the best results and generates 12. W. Dong, L. Zhang, R. Lukac, and G. Shi, IEEE Trans. Image
much sharper edges than method [8]. Process. 22, 1382 (2013).
In conclusion, we propose a new demosaicing model for 13. W. Dong, L. Zhang, G. Shi, and X. Li, IEEE Trans. Image Process. 22,
DoFP images. This model is based on sparse representation and 1620 (2013).
keeps the correlation among different polarization channels. 14. S. Foucart and H. Rauhut, A Mathematical Introduction to
Compressive Sensing (2013), Vol. 1.
Adaptive sub-dictionary is introduced to generate more sparse 15. J. Zhang, H. Luo, R. Liang, W. Zhou, B. Hui, and Z. Chang, Opt.
and accurate coding coefficients. Synthetic and real DoFP Express 25, 2391 (2017).
images are used to evaluate the interpolated performance. 16. W. Dong, L. Zhang, G. Shi, and X. Wu, IEEE Trans. Image Process.
The experimental results demonstrate that our proposed 20, 1838 (2011).