Proceedings of the 19th International Conference on Digital Signal Processing
An EM- and wavelet-based multi-band image
restoration approach
Yifan Zhang
Shaanxi Key Laboratory of Information Acquisition and Processing
School of Electronics and Information, Northwestern Polytechnical University
Xi’an 710129, P. R. China
Email: yifanzhang@nwpu.edu.cn
Abstract—This work presents a restoration approach with two
available observations of the same scene, each with specific spatial
and spectral resolutions. The proposed restoration approach
is based on a multi-band image restoration scheme employing
Expectation-Maximization (EM) algorithm and implemented in
wavelet domain. It can be easily applied in hyperspectral and
multispectral image fusion and restoration to acquire image with
both high spatial and spectral resolutions. Simulation experiment
is employed to validate the proposed approach. A performance
comparison is made with its counterparts in image domain.
Index Terms—Expectation-Maximizition (EM) algorithm, hy-
perspectral image, multipectral image, restoration.
I. I NTRODUCTION
With the rapid development in remote sensing technology
during the last two decades, multispectral (MS) data sets
acquired by optical remote sensors have grown steadily to
hyperspectral (HS) ones, which employ hundreds of contigu-
ous spectral bands to capture and process spectral information
over a range of wavelenghts, compared to the tens of discrete
spectral bands used in MS images [1]. Nevertheless, due to
some physical and practical limitations, images with both high
spatial and spectral accuracy are not available. In most cases,
the spatial resolution of HS images is lower than that of MS
images [2]. The requirement of both high spatial and spectral
resolutions in practical applications greatly inspires research
on spatial and spectral restoration techniques.
There exists vast literature on image restoration, in most
of which the observation model for HS image is inversed
based on the knowledge of its point spread function (PSF),
and sometimes using MS image with high spatial resolution as
auxiliary. In [3], an Expectation-Maximization (EM) algorithm
for image restoration is proposed, in which a deconvolution
step and a denoising step are applied separately. Duijster et
al extended this work to MS image restoration [4]. In our
previous work [5], fusion technique is combined with EM-
based restoration technique for HS image restoration with a
high spatial resolution MS image as auxiliary. Spectral restora-
tion of the MS observation by inverting its corresponding
observation model is another solution. In [6], the spectrally re-
stored image is obtained using a Wiener estimation. However,
the performance is quite limited. It is also feasible to apply
the restoration schemes of [4] and [5] in spectral domain to
spectrally restore the MS observation, and using HS image
978-1-4799-4612-9/14/$31.00 © 2014 IEEE
20-23 August 2014
with high spectral resolution as an auxiliary is also possible.
To eliminate limitation of the separate spatial and spectral
restoration techniques that the observation model (including
the degradation function and model noise) for the one denoted
as auxiliary is not appropriately treated, a spatial and spectral
combined complete restoration scheme is presented in [7], in
which both spatial and spectral restoration are employed and
implemented alternately.
In this paper, a multi-band image restoration approach is
proposed, concerning two observations of the same scene with
specific spatial and spectral resolutions. The EM algorithm
is employed, in which the restoration is divided into two
separated steps, and the restored image is estimated iteratively.
Particularly, the approach is implemented in wavelet domain
considering its excellent performance in multi-scale analysis.
In the simulated experiments, the proposed restoration ap-
proach is validated and also compared with its counterparts
in image domain. And the proposed approach can be easily
applied in HS and MS image fusion and restoration technique.
II. T HE EM- AND WAVELET- BASED RESTORATION
APPROACH
A. Problem description
By presenting each image in wavelet domain and also in
band-interleaved-by-pixel lexicographical notation, the obser-
vation model for an observed image with specific spatial and
spectral resolution can be expressed as
M = WS + N (1)
where W reflects the spatial and spectral degradation of the
observation M from the ideal image S and N represents
additive Gaussian noise with zero mean and covariance matrix
CN .
It is notable that the W is obtained from the corresponding
degradation function in image domain. With undecimated
wavelet transform using the orthogonal wavelet families that
we apply in this paper, the degradation function holds the same
for both spatial and wavelet domain. The noise covariance in
each wavelet detail channel is in general a scaled version of the
input image noise covariance, where the scaling factors depend
on the wavelet filter coefficients. By using the orthogonal
wavelet families, the noise covariance in all the wavelet detail
channels is equal to the input image noise covariance.
617 DSP 2014
Proceedings of the 19th International Conference on Digital Signal Processing 20-23 August 2014

In this paper, an optimal description of a scene S will be where 𝔼[⋅]represents the expectation value and C⋅,⋅ the covari-
estimated from two available observations: observation M1 ance matrix defined as:
(HS observation) with low spatial resolution high spectral
resolution and observation M2 (MS observation) with high C𝝉 ,𝝂 = 𝔼[(𝝉 − 𝔼[𝝉 ])(𝝂 − 𝔼[𝝂])𝑇 ] (7)
spatial resolution low spectral resolution. It is notable that we and specifically
will assume that all images are equally spatially sampled at a
grid of 𝑁 pixels which is sufficiently fine to reveal the spatial C𝝉 = C𝝉 ,𝝉 = 𝔼[(𝝉 − 𝔼[𝝉 ])(𝝉 − 𝔼[𝝉 ])𝑇 ]. (8)
resolution of M2 , and all images are also spectrally sampled
Since the S-dependent part of this expression is linear in
at a grid of 𝑃 spectral bands which is sufficiently fine to reveal
X, finding the 𝑄-function or the expectation of (4) comes
the spectral resolution of M1 . Accordingly, the restored result
down to finding the expectation of X, conditioned on the
Ŝ will be with the same spatial resolution as M2 and the same
observations Mobs and Maux as well as Ŝ(𝑘−1) . We will denote
spectral resolution as M1 .
this expectation as X̂(𝑘) , so that the final expression for the
B. EM-based restoration scheme 𝑄-function becomes:
The EM algorithm is designed as an iterative procedure of
𝑄(S, Ŝ(𝑘−1) ) =
obtaining MAP/MPLE estimate of a parameter by maximizing
1
the complete-data likelihood function [3]. Two alternate steps 𝜶 − (S − X̂(𝑘) )𝑇 C−1
Nobs (S − X̂
(𝑘)
)
(E-step and M-step) are involved at each iteration. The key 2
1
concept in the EM-based image restoration scheme is that − (S − 𝝁S∣Maux )𝑇 C−1
S∣Maux (S − 𝝁S∣Maux ). (9)
2
the observation model is inverted by performing the two-step
algorithm (deblurring and denoising).To accomplish this, the To solve this, an estimate of X is required. The pdf
observation model is decomposed as: of X conditioned on the previous estimate of S and both
observations can be shown as:
M = WX + N′′ (2)
X = S + N′ . (3) 𝑝(X∣Mobs , Ŝ(𝑘−1) , Maux )
∝ 𝑝(Mobs ∣X)𝑝(Ŝ(𝑘−1) ∣X)𝑝(X∣Maux ) (10)
In this way, the noise is decomposed into two independent
parts N′ and N′′ , with WN′ + N′′ = N. with
The splitting up leaves the option to divide the originally
assumed Gaussian white noise N (with diagonal covariance 𝑝(Mobs ∣X) = 𝜙(Wobs X, CNobs − Wobs CNobs Wobs T
)
matrix CN ) into two parts. Choosing N′ to be white with 𝑝(Ŝ(𝑘−1) ∣X) = 𝜙(X, CNobs )
𝑝(N′ ) = 𝜙(0, CN ) facilitates the denoising problem (3). 𝑝(X∣Maux ) = 𝜙(𝝁Ŝ(𝑘−1) ∣Maux , CŜ(𝑘−1) ∣Maux + CNobs ).
However, W colors the noise so that N′′ becomes colored with
𝑝(N′′ ) = 𝜙(0, CN − WCN W𝑇 ). (2) and (3) are iteratively Thus, an estimate of the expectation of X can be obtained
solved using the EM-algorithm (see [4] for more information). as:
In the EM-based restoration scheme, both observations X̂(𝑘) = Z + C(Mobs − Wobs Z) (11)
could be restored independently. The restored image could be
obtained by spatial restoration of M1 , or by spectral restora- with
tion of M2 , independently. While restoration performance can C = BWobs
T
[CNobs + Wobs (B − CNobs )Wobs T −1
]
be greatly improved by using the other one as auxiliary. For the −1 −1 −1
B = [CNobs + (CŜ(𝑘−1) ∣Maux + CNobs ) ]
convenience of description, we denote the one to be restored
as Mobs , and the other one as auxiliary image Maux . Z = B[C−1
Nobs Ŝ
(𝑘−1)
+ (CŜ(𝑘−1) ∣Maux + CNobs )−1 𝝁Ŝ(𝑘−1) ∣Maux ].
1) E-step: By using Maux as an auxiliary and making
𝜶 represent all items independent on S, the complete log- 2) M-step: In M-step, the estimate of S is updated by
likelihood can be rewritten as maximizing the 𝑄-function in (9), which leads to

log(𝑝(S, X∣Mobs , Maux )) (4) Ŝ(𝑘) = arg max 𝑄(S, Ŝ(𝑘−1) )

S
1 𝑇 −1
= 𝜶 − (S − X) CNobs (S − X) = CŜ(𝑘−1) ∣Maux (CŜ(𝑘−1) ∣Maux + CNobs )−1 X̂(𝑘)
2
1 𝑇 +CNobs (CŜ(𝑘−1) ∣Maux + CNobs )−1 𝝁Ŝ(𝑘−1) ∣Maux(12)
.
− (S − 𝝁S∣Maux ) C−1 S∣Maux (S − 𝝁S∣Maux )
2
C. Spatial and spectral restoration in wavelet domain
where we have assumed that S and Maux are jointly nor-
mally distributed, so that the pdf 𝑝(S∣Maux ) is also a normal When M1 is denoted as Mobs , M2 as Maux , a spatial
𝜙(𝝁S∣Maux , CS∣Maux ) with conditioned mean and covariance restoration of M1 is implemented. The obtained Ŝ is literarily
matrix expressed as: a spatially restored result of M1 with the higher spatial
resolution observation M2 as an auxiliary (SpaR). When M2
𝝁S∣Maux = 𝔼[S] + CS,Maux C−1
Maux (Maux − 𝔼[Maux ]) (5) is denoted as Mobs , M1 as Maux , a spectral restoration of
CS∣Maux = CS − CS,Maux C−1 𝑇
Maux CS,Maux (6) M2 is implemented. The obtained Ŝ is literarily a spectrally

978-1-4799-4612-9/14/$31.00 © 2014 IEEE 618 DSP 2014

Proceedings of the 19th International Conference on Digital Signal Processing 20-23 August 2014

restored result of M2 with the higher spectral resolution ∙ SAM: The averaged Spectral Angle Mapper (SAM) is
observation M1 as an auxiliary (SpeR). used as a measurement of spectral distortion between the
Either spatial or spectral restoration focuses on only one restored and reference images. The SAM of two spectral
of the observation models. As a result, either knowledge vectors (S𝑛 and Ŝ𝑛 ) at a given spatial position 𝑛 is
about spatial degradation W1 or knowledge about spectral defined as
degradation W2 would not be utilized. And only noise in the
considered observation model is appropriately treated, while ⟨S𝑛 , Ŝ𝑛 ⟩
SAM(S𝑛 , Ŝ𝑛 ) = arccos( ). (14)
the noise in the other one is simply ignored. For the purpose of ∥S𝑛 ∥2 ⋅ ∥Ŝ𝑛 ∥2
equal and appropriate usage of the two observations and their
corresponding observation models, spatial restoration of the It can be measured in either degrees or radians. This
observation M1 and spectral restoration of the observation M2 value is averaged over all pixels of the image. The smaller
can be implemented alternately, which is literarily a combined values of SAM, the less spectral distortion. In this paper,
spatial-spectral restoration approach (SSR). we measure the SAM in degrees.
In this paper, the restoration strategies mentioned above are ∙ UIQI: A universal image quality index (UIQI) which has
implemented in wavelet domain rather than image domain[5], been widely used for image similarity evaluation and was
[7]. Compared to direct image domain representation, the also applied to validate restoration techniques [8]. UIQI
wavelet transform spatially decorrelates the pixels allowing of two images (𝐴 and 𝐵) is defined:
for a simplified and more accurate modeling, besides, it also
4𝜎𝐴𝐵 𝜇𝐴 𝜇𝐵
allows for a scale- and orientation-specific estimation of the 𝑄 = 2 + 𝜎 2 )(𝜇2 + 𝜇2 )
model parameters. Restoration performance improvement can (𝜎𝐴 𝐵 𝐴 𝐵
be expected due to estimation in a more controlled manner in 𝜎𝐴𝐵 2𝜇𝐴 𝜇𝐵 2𝜎𝐴 𝜎𝐵
= ⋅ ⋅ 2 2 . (15)
wavelet domain. 𝜎𝐴 𝜎𝐵 𝜇2𝐴 + 𝜇2𝐵 𝜎𝐴 + 𝜎𝐵

III. E XPERIMENTAL RESULTS AND ANALYSIS This quality index models any distortion as a combination
of three different factors: loss of correlation, luminance
To validate and evaluate the restoration performance ob- distortion, and contrast distortion. The dynamic range of
jectively and quantitatively, simulated experiments with a 𝑄 is [−1, 1] and the best value 1 is obtained if 𝐴 = 𝐵.
reference are employed in this paper. A set of Landsat image When applying this index to a multiband image, it is
(with size of 256×256×6) is adopted as the reference S, with applied band-by-band and averaged over all bands.
the spatial resolution of 30m. Spatial-invariant blurring using
a 2D Gaussian low-pass filtering (with standard deviation B. Results and analysis
𝜎𝑠 = 1.2) is applied to S, and Gaussian white noise is also
added, to simulate the observation M1 . To construct M2 The restored images of different approaches are shown
which is a spectral degradation of S, spectral vector of S at in Fig. 1 (displayed with band 3, 2, 1 assigned as R, G,
each spatial position is smoothed by an 1D Gaussian low- B channels respectively), as well as the reference and both
pass filter with standard deviation of 0.6 and Gaussian white observations. All the implemented restoration approaches are
noise is added. The three restoration strategies (SpaR, SpeR capable of restoring observed images effectively, obvious
and SSR) are implemented in wavelet domain and compared differences among their results can hardly be noticed only by
with their counterparts in image domain. For EM algorithm, observation. In order to compare their restoration performance
30 iterations are employed. more objectively and quantitatively, the three evaluation in-
dex values for restoration approaches after convergence are
A. Evaluation index listed in Table I. It can be denoted that the restored results
The differences among the restored results are usually acquired based on wavelet implementation outperform the
difficult to be measured solely based on observation, especially ones acquired in image domain implementation a little. This
when the restored images are multi-band. Objective and quan- improvement can be attributed to a scale- and orientation-
titative analysis can benefit to a comprehensive evaluation. specific estimation strategy as well as more effective noise
Various image quality indices have been developed. Three such treatment in wavelet domain.
evaluation indices are employed in this paper: TABLE I
∙ SNR: The signal to noise ratio (SNR) in decibels is E VALUATION INDICES FOR COMPARED APPROACHES
a direct index to compare the restored image to the
reference one. For multi-band images, it can be calculated SpaR SpeR SSR
band-by-band and also globally averaged. In this paper, image wave image wave image wave
SNR 31.77 32.16 31.79 32.08 32.44 32.62
a globally averaged SNR is employed. SAM 1.3782 1.3088 1.3572 1.3153 1.2687 1.2356
∑ 2 UIQI 0.9921 0.9926 0.9919 0.9922 0.9929 0.9931
S
SNR(S, Ŝ) = 10log10 ∑ (13)
(S − Ŝ)2

978-1-4799-4612-9/14/$31.00 © 2014 IEEE 619 DSP 2014

Proceedings of the 19th International Conference on Digital Signal Processing 20-23 August 2014

IV. C ONCLUSION
The problem discussed in this paper, is to estimate an
optimal description of a scene with two observations available,
each obtained at specific spatial and spectral resolution scales.
To settle this problem, a restoration approach is proposed
based on a multi-band image restoration scheme employing
EM algorithm. The proposed restoration approach is imple-
mented in wavelet domain to allow a scale- and orientation-
specific estimation of model parameters. Simulation experi-
(a) Reference ment is employed to validate the proposed approach and com-
pare with its counterparts in image domain. The experimental
results illustrate its improved restoration performance over
image domain implementation. HS and MS image fusion and
restoration would be a potential and promising application area
of the proposed approach.
ACKNOWLEDGMENT
This work is partially supported by the National Natural
Science Foundation of China under Grant 61171154 and
61101188, and the Fundamental Research Funds for the Cen-
tral Universities under Grant 3102014JCQ01056.
(b) M1 (c) M2
R EFERENCES
[1] C.-I. Chang, Hyperspectral imaging: techniques for spectral detection and
classification. Kluwer Academic Publishers, 2003.
[2] R. Gomez, A. Jazaeri, and M. Kafatos, “Wavelet-based hyperspectral and
multi-spectral image fusion,” Proc. SPIE, vol. 4383, pp. 36–42, 2001.
[3] M. A. T. Figueiredo and R. D. Nowak, “An EM algorithm for wavelet-
based image restoration,” IEEE Trans. Image Process., vol. 12, no. 8, pp.
906–916, Aug. 2003.
[4] A. Duijster, P. Scheunders, and S. De Backer, “Wavelet-based em algo-
rithm for multispectral-image restoration,” IEEE Trans. Geosci. Remote
Sens., vol. 47, no. 11, pp. 3892–3898, Nov. 2009.
[5] Y. Zhang, A. Duijster, and P. Scheunders, “A Bayesian restoration
approach for hyperspectral images,” IEEE Trans. Geosci. Remote Sens.,
vol. 50, no. 9, pp. 3453–3462, Sep. 2012.
(d) SpaR-image (e) SpaR-wave
[6] N. Shimano, “Recovery of spectral reflectances of objects being imaged
without prior knowledge,” IEEE Trans. Image Process., vol. 15, no. 7,
pp. 1848–1856, July 2006.
[7] Y. Zhang, “A spatial and spectral combined image restoration approach,”
Proc. 2012 IEEE Int. Geosci. Remote Sens. Symp., pp. 3509–3512, Jul.
2012.
[8] Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparative
analysis of image fusion methods,” IEEE Trans. Geosci. Remote Sens.,
vol. 43, no. 6, pp. 1391–1402, June 2005.

(f) SpeR-image (g) SpeR-wave

(h) SSR-image (i) SSR-wave

Fig. 1. Restored results of compared approaches.

978-1-4799-4612-9/14/$31.00 © 2014 IEEE 620 DSP 2014