IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO.
3, MARCH 2011
Design of Interchannel MRF Model for Probabilistic
Multichannel Image Processing
Hyung Il Koo, Member, IEEE, and Nam Ik Cho, Member, IEEE
Abstract—In this paper, we present a novel framework that
exploits an informative reference channel in the processing of
another channel. We formulate the problem as a maximum a
posteriori estimation problem considering a reference channel and
develop a probabilistic model encoding the interchannel correla-
tions based on Markov random fields. Interestingly, the proposed
formulation results in an image-specific and region-specific linear
filter for each site. The strength of filter response can also be con-
trolled in order to transfer the structural information of a channel
to the others. Experimental results on satellite image fusion and
chrominance image interpolation with denoising show that our
method provides improved subjective and objective performance
compared with conventional approaches.
Index Terms—Chrominance denoising, chrominance interpola-
tion, Markov randiom field (MRF) model, multichannel image pro-
cessing, satellite image fusion.
I. INTRODUCTION
T HE correlations of neighboring pixels and the relation-
ship between the channels are basic ingredients for de-
veloping many image processing algorithms. For example, the
spatial correlations are encoded in the form of prior probability
of an image, and the priors are used in a variety of applications
under the maximum a posteriori (MAP) frameworks [1]–[10].
Examples include image interpolation, image denoising, image
inpainting, image deblurring, and so on. On the other hand, the
correlations across channels are usually handled in determin-
istic ways [11]–[16]. Although some deterministic approaches
effectively exploit the interchannel correlations, they assumed
true pixels in a channel image, and thus it is not straightforward
to extend them to the processing of noisy observations. More-
over, the deterministic nature hinders the use of the channel
correlations with other cues. Hence, we present a new prob-
abilistic approach that handles the channel correlations in the
form of conditional probability, i.e., the probability of a channel
image given a reference channel. This means that we can use
the channel correlations in a similar manner as the spatial cor-
relations.
Manuscript received June 16, 2009; revised January 20, 2010, May 15, 2010;
accepted August 18, 2010. Date of publication September 23, 2010; date of cur-
rent version February 18, 2011. This work was supported by the Basic Science
Research Program through the National Research Foundation of Korea (NRF)
funded by the Ministry of Education, Science and Technology under Contract
20100001961. The associate editor coordinating the review of this manuscript
and approving it for publication was Prof. Peter C. Doerschuk.
The authors are with the Department of Electrical Engineering and Computer
Science and INMC, Seoul National University, Seoul 151-744, Korea (e-mail:
hikoo@ispl.snu.ac.kr, nicho@snu.ac.kr).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2010.2073473
1057-7149/$26.00 © 2011 IEEE
601
A. Spatial Correlations
There have been a number of image priors that encode the
spatial correlations [1]–[3]. In designing the priors, the most
popular approach is based on Markov random fields (MRFs).
MRFs are undirected graphical models, where the vertices of a
graph represent random variables that correspond to image mea-
surements and the edges indicate the factorization structure of
image priors. Precisely, from the Markov random fields—Gibbs
random fields (MRFs-GRFs) equivalence, the probability of an
image can be expressed as a product of potentials over
maximal cliques
(1)
where is the image region corresponding to the clique [3],
[17]. In the design of , many early prior models were
based on the smoothness constraints of adjacent pixels, which
could be achieved by enforcing the magnitude of spatial deriva-
tives to be small [1]–[3]. Recently, it has been shown that image
priors can be improved by considering the sparse nature of spa-
tial derivatives, and these sparse priors have been successfully
applied to several applications such as reflection removal and
deblurring [5], [6].
Although such prior models provide good performance in
many applications, they lack theoretical justification and have
limitations in handling higher order statistics [7]. Hence, the
priors learned from training data were proposed in order to cap-
ture the rich and higher order statistics of natural images [4], [7],
[8], [10]. For example, the filters, random fields and maximum
entropy (FRAME) model constructs image priors from learned
filters and learned potential
(2)
where is the filtered image of by the filter
is the filter response at the site , and is
the partition function [4], [8], [10]. Although their framework
provides rich and generic prior models for images, they suffer
from huge computational burden in learning and inference.
B. Channel Correlations
Although the channel correlations have been considered in
many applications such as demosaicking, chrominance inter-
polation, colorization, and compressions [11]–[15], [18], [19],
there has been little research on the development of probabilistic
models that encode the channel correlations. Since probabilistic
602
Fig. 1. IKONOS relative spectral response.
approaches provide several advantages over deterministic
methods, we develop a probabilistic model considering the
channel correlations. Let us denote a multichannel image as
, where is the number of channels and
is the th channel image . Although color
images consisted of red (R), green (G), and blue (B) channels
are most common multichannel images, there are a number
of other cases. For example, satellite images from IKONOS,1
which is the first commercial high-resolution satellite launched
in 1999, consist of panchromatic (PAN), B, G, R, and near-in-
frared (NIR) channels . The spectral response of
each channel can be found in Fig. 1. In multichannel images,
the resolution and/or signal-to-noise ratio (SNR) often varies
from one channel to another due to the asymmetry of channels
(sensors). The resolution of PAN images is 4 higher than that
of the spectral images (R, G, B, and NIR) in satellite images
[20], and NIR images suffer much less from noise in low-light
and night-vision applications [21]. Also, the resolution of
luminance images is 2 higher than that of the chrominance
images in many image/video standards [22].
Since there are strong correlations between channels, an in-
formative channel may help the processing of another channel,
that is, high-resolution channel images can help the interpola-
tion of other channel images, and high SNR images can help
the denoising of others. Based on these observations, we pro-
pose a general framework that allows the use of an informative
reference channel in the processing of another. We formulate
the problem as a MAP estimation problem considering a refer-
ence channel. Unlike conventional single-channel methods, the
problem requires the modeling of , where is an
image under processing and is its reference.
In order to determine whether is a probable one given its
reference , we try to find the relation between pixels in
and pixels in . This process is based on the color image for-
mation model [11], [16], [20] and minimum mean-squared error
(MMSE) estimator [12]. The error in this model indicates the
feasibility of the model, i.e., is a probable image only when
the error is small. Therefore, we consider the errors as clique
potentials and construct a probabilistic model of by
using the MRF-GRF equivalence. Interestingly, the result can
be considered as a special case of (2): an image-specific and re-
gion-specific filter is given for each site, and the channel correla-
tions are effectively exploited by enforcing the filter response to
1[Online]. Available: http://www.geoeye.com
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 3, MARCH 2011
be small. Although the channel correlations have been exploited
in several ways, our method is the first approach that models the
correlations based on MRFs, and our method provides a general
framework considering a more informative channel in the pro-
cessing of the others. We have applied our method to several
applications including satellite image fusion and chrominance
image interpolation with denoising. Experimental results show
that the proposed method improves the performance over the
conventional methods.
The remainder of this paper is organized as follows. We first
explain our interchannel MAP-MRF framework in Section II
and present a new probabilistic channel model in Section III.
The proposed conditional probability is presented in Section IV,
and experimental results for the applications will be given.
II. MAP-MRF FRAMEWORK CONSIDERING
THE CHANNEL CORRELATIONS
Here, we introduce a new MAP-MRF formulation that ex-
ploits the channel correlations. We first explain a conventional
MAP-MRF framework for the single-channel case and then in-
troduce a new formulation considering a reference channel. Un-
like conventional methods requiring image priors, our formula-
tion requires the modeling of conditional probability .
We also discuss merit and demerit of our method compared to
a more general model.
A. Conventional MAP-MRF Framework
The MAP-MRF framework has received much attention in
image processing and computer vision [8], [17]. In this frame-
work, we estimate a true image from its corrupted observation
by maximizing its posterior probability . Let be the
estimate of , and then
(3)
where is called a data term and is a prior term.
When both and are convex, the
MAP inference can be done in a relatively simple way and the
global optimality is guaranteed. Otherwise, the gradient-decent
algorithm or a more complicated method such as the itera-
tive reweighted least squares (IRLS) process may be adopted
[5]–[8], however, the global optimum is no longer guaranteed.
B. MAP-MRF Framework Considering the Channel
Correlations
In order to develop a method that exploits an informative
channel, we assume that a reference image is available in
the processing of the th channel . In this framework,
we can estimate a true image from its degraded observation
and its reference by maximizing the posterior proba-
bility . Let be the estimate of , and then
(4)
(5)
KOO AND CHO: DESIGN OF INTERCHANNEL MRF MODEL FOR PROBABILISTIC MULTICHANNEL IMAGE PROCESSING
where the Bayes’ rule is used. Since is the observation of
, and this can be further reduced to
(6)
from the Markovian property. The first part of the cost function
comes from observation models that depend on applications [3],
[8]. Hence, we need to design for a new MAP-MRF
estimator.
C. Relation to the Joint Probability
Although we focus on the development of conditional prob-
ability by assuming that a noise-free reference is available,
it should be noted that the joint probability is a
more powerful model. From the joint probability, we can get
the marginal probability and the conditional probability
. Also, it enables joint estimation. In order words, we
can estimate and simultaneously from their degraded
observations ( and ) by maximizing the posterior prob-
ability of them as
(7)
(8)
(9)
Here, the Bayes’ rule and the Markovian property are used, and
the subscripts “ ” indicate that they are the results of the joint
estimation (we use this notation in order to differentiate them
from the results of other estimators). However, the design of
is believed to be a very difficult task, since we have
to consider the spatial correlations and the channel correlations
simultaneously. It is well known that even the learning of spatial
correlations (i.e., the learning of the marginal probability) is a
difficult and very time-consuming task [8], [10]. Moreover, its
extension to multichannel images having more than two chan-
nels (i.e., ) is not straightforward.
Hence, rather than developing the joint probability and opti-
mizing equation (9), we try to develop a relatively simple model
and focus on the optimization of (6). Fortunately, this approach
is effective in many interesting cases. Almost noise-free refer-
ence images are available in satellite image fusion application
[20] and night-vision applications [21], so that (6) is a suffi-
cient formulation. Even though noise-free reference images are
not available, we can develop a suboptimal method that can
handle nonideal references. For the explanation of our subop-
timal method, let us rewrite (9) as
(10)
When the th channel is more informative (e.g., high resolution
and/or high SNR) compared with the th channel,
603
will show a sharper peak than other terms. Therefore, the max-
imum of (10) will occur when is close to the optimal solution
of the single-channel method, that is,
(11)
Moreover, by assuming this approximation is accurate, we can
estimate the th channel image as
(12)
(13)
Since (11) is a conventional (single-channel) problem and (13)
is the same as (6), our method based on (6) can address the joint
estimation problem, although it may not be optimal. Note that
this approximation holds only when the th channel is much
informative, and we have to solve (13) after (11).
Fortunately, there are many algorithms showing improved
performance with less complexity compared to MRF-based
methods, and we can use them for the first subproblem (11).
To be specific, wavelet-based denoising methods such as [23]
show better denoising performance than MRF-based methods
[8]. Also, it is known that the edge-directed image interpolation
methods outperform MRF-based methods [12], [24]. In the
experimental section, we have used this greedy method in order
to handle nonideal references, and the results show that cor-
rupted references can also improve the performance of another
channel processing as long as it is informative. In summary, (6)
is an effective formulation that can cover many interesting ap-
plications, and the design of conditional probability
allows the estimation.
Finally, it is worth noting that we can construct the joint prob-
ability by multiplying our conditional probability with the con-
ventional image priors [3], [5], [8]
(14)
However, we have found that this model is not a practical one
due to its computational complexity, which will be discussed in
Section IV.
III. CHANNEL MODEL
Here, we review conventional channel models and propose
a new probabilistic model in order to represent the probability
of given its reference . We assume that the resolution of
each channel image is the same (their observations may have
different resolutions) and they are defined on , which is a set of
sites (pixels) as shown in Fig. 2. Also, we denote the positional
vector at the site as , and indicates the
pixel value of the th channel image at .
A. Color Image Formation Model
Although several color image formation models were devel-
oped in different contexts [11], [14], [16], [20], their results are
basically the same, that is, a pixel value is modeled as a linear
function of the other one in another channel. In the case of RGB
604 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 3, MARCH 2011

as sigmoid functions that have transition zones. Therefore, we

assume that and can be approximated to linear func-
tions in a local neighborhood. Precisely, we denote the local
neighborhood of a site as

(18)

as shown in Fig. 2 and denote the first-order approximations

of and on as and respectively. Note
that we can represent a first-order approximation around as
for some and . Finally, our
Fig. 2. We assume that two images I and I are defined on P . As illustrated, model can be represented as
p indicates a site (pixel) and c is a neighborhood around p. In our method, a
filter w , whose support is c , is assigned to each site p. The filter is computed
from I and it has structural information of I . Hence, by minimizing the filter (19)
response of I at p, we can force the structure of I to be consistent with its
reference. for all , where is a zero-mean Gaussian noise. Since
the variance of depends on the statistical property of an input
image, we denote its variance as .
channels [11], the model for color images is derived in the con-
text of the “Mondriaan world”, i.e., the world consists of Lam- IV. PROPOSED MODEL
bertian nonflat surface patches with fixed light direction . This
Here, we present a probabilistic model encoding the inter-
means that the RGB vector at a 3-D point in the space is given
channel correlations. The model is based on the observation
by
that the errors in (19) indicate the feasibility of a pair,
and we design the probability using the MRF-GRF equivalence.
(15) Also, we explain the way to use the derived probability in image
processing tasks.

where is the normal vector at and A. MRF Formulation

is the albedo of material that changes according to channels. For the MRF formulation, we consider the local neighbor-
Hence, the ratio of channel values at a pixel position , which hood as a clique, and define its clique potential as
corresponds to the 3-D point , is expressed as

(16)
(20)
Since the luminance-chrominance representation (e.g., YCbCr,
where is a vector representing the parameters of
YUV) is simply an affine transformation of RGB representation,
and , and is the number of sites (pixels) in , i.e.,
the relationship of and in such color spaces is given
. From the MRF-GRF equivalence, the probability of
by
, given and , can be expressed as
(17)
(21)
where and are two functions defined on the image do-
main . Since the estimation of and is a severely where is a partition function. If we consider as a random
ill-posed problem, several assumptions on two functions were field, we can get
posed in previous work. In chrominance interpolation and de-
mosaicking, two functions are modeled as piecewise constant (22)
functions by assuming that albedo is a constant in the same ob-
ject (material). Under the assumption, we can estimate and
from some true pixel values, and missing pixel values can which is intractable due to infinitely many instances of . In
be computed from them. However, such an approach is appli- order to make it tractable, we assume that the prior is non-in-
cable only to limited applications, because the parameter esti- formative (i.e., and shows
mation is sensitive to noise. a very sharp peak at it optimal estimate , so that
. Here, the estimate can be found
B. Proposed Probabilistic Channel Model by solving least squares problems. In summary, the conditional
probability is given by
In many cases, and are slowly varying within an
object. Even if they may experience abrupt changes at object (23)
boundaries, these changes are not modeled as step functions but
KOO AND CHO: DESIGN OF INTERCHANNEL MRF MODEL FOR PROBABILISTIC MULTICHANNEL IMAGE PROCESSING 605

evaluation of is . Fortunately, the

amount of memory and computation can be dramatically re-
duced using a simple approximation

(28)

Fig. 3. Three image patches from different channels. (a) I . (b) I . (c) I .
Since corresponding pixels in I and I show a linear pattern, 8(c ) is small, (29)
indicating that I is a probable one when I is given. On the other hand, 8(c )
corresponding I and I is large and we can expect that I is an unlikely patch
when I is its reference. (30)

where represents a vector corresponding to the th row.

where We denote the approximate potential as , which can also
be expressed using the convolution notation
(24)

Since there is no unknown parameter in (23) and is a constant, (31)

it can be directly used in (6).
(32)
As can be seen in Fig. 3, our method naturally prefers a sim-
ilar structure to the reference channel . Precisely, the energy
of is about 30 times higher than that of , which means where is the filter corresponding to .
. Although we do not explicitly con- Since the computation of requires expensive operations, we
sider edges or object boundaries in our model, this property nat- have to compute in the initialization step and reuse
urally transfers the edge structure of a reference image to the them during the optimization. Unlike other learned priors where
current one. the responses of several linear filters are considered for each
site, the proposed method provides a single filter for each
site. In addition, the filter response is just desired to be small.
B. Approximations Memory requirement for this approximation is and
computational complexity is also . Intuitively, the
Here, we approximate in (24) to a much computation- approximation in (30) is to replace the average error of
ally efficient form. For the goal, we first represent samples with one of them. Though this approximation
using matrix notations as may be poor in some cases, it is performed a number of times,
and the poor approximation errors will be averaged out in the
(25) final result, i.e., (see (23) and the
cost function (35) in Section IV-C).
Finally, for the intuitive understanding of our method, let us
where is a vector corresponding to the pixels of
introduce a simple example for . Imagine a block of a
on , and is the matrix corresponding reference channel whose values are
to a clique in the image . By using the standard matrix
computation techniques, we can get , where is
the pseudo inverse of . Then, can be rewritten as
(33)

(26)

Then, the derived filter at the center site is given by

(27)
(34)

where is a identity matrix and

. This can be im-
plemented by using the precomputed masks corresponding Here, the filter possesses the local structure of a reference
to . Memory requirement for this process is channel around , and we can obtain the same structure in
, and the computational complexity for a single by minimizing .
606
TABLE I
SUMMARY OF THE PROPOSED METHOD
C. Inference
Using energy-based representation, our cost function in (6) is
given by
(35)
(36)
where and is a constant that
controls the weights of two complementary terms. Since the ap-
proximate level of observation noise may be known a priori in
many applications [8], the data term can be normalized. How-
ever, in the second term shows image-dependent
variance , and the normalization of the second term is
not a simple task. Therefore, we have to determine a weighting
constant empirically [8], [25], [26]. The choice of and the
effects of will be discussed in the experimental section.
Since the joint probability is the product of
and spatial priors , the cost function in (9) can
be well defined by using our conditional probability. However
the minimization of (9) is not straightforward. Note that we
make the evaluation and inference tractable by using the pre-
computed masks, and this precomputation assumes fixed .
We have summarized our method in Table I.
V. EXPERIMENTAL RESULTS
Our method finds the estimate by minimizing (36). In
experiments, we set (equivalent to a 7 7 neighborhood
system in Fig. 2) for 512 512 and 768 512 images.
A. Single-Channel Denoising in Color Images
In order to demonstrate the performance of our method and
analyze the effects of parameters, we have conducted denoising
experiments on the Kodak Photo CD images which are widely
used in demosaicking researches. First, in order to know the ef-
fects of parameters, we have assumed that one channel is noise-
free and other channels are corrupted with zero-mean Gaussian
noise whose variance is . Note that this situation is similar to
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 3, MARCH 2011
practical cases [21]. By assuming homogeneous and pixel-wise-
independent Gaussian noise, the data term in (36) is given by
(37)
(38)
and our cost function (36) becomes
(39)
Since the cost function is convex and the analytic derivative of
the function can be easily derived from the precomputed ,
the conjugate gradient algorithm can be easily implemented and
guarantees a global optimum [27]. In the optimization, we do
not explicitly impose the constraints on the range of
(such as nonnegativity). Rather, these constraints were imposed
by clipping reconstructed values.
In experiments, we find a constant that
shows the largest PSNR improvements in our dataset
and test our method for several weighting parameters:
, and . The
denoising performance of B-channel images given R-channel
images ( B-channel, R-channel) when are
summarized in Fig. 4. From the graph, we can find several
properties of our method. First, the PSNR gains of our method
decrease as increases. Second, a large weighting
constant (i.e., putting more weights on the term consid-
ering the channel correlations) yields better results for small
. Since indicates the strength of channel cor-
relations, these two results are not surprising. Third, although
it may not be an optimal choice, the method using fixed
yields good performance over the wide range of parameters
(e.g., ) and the improvement over a
single-channel image algorithm (BLS-GSM algorithm in [23])
is quite noticeable (about 5 dB). Finally, we can observe that
the performance of a single-channel algorithm also decreases
as increases. Since the BLS-GSM algorithm does not
exploit the channel correlations, we believe that this indicates
KOO AND CHO: DESIGN OF INTERCHANNEL MRF MODEL FOR PROBABILISTIC MULTICHANNEL IMAGE PROCESSING
Fig. 4. Denoising performance of B-channel images given ideal R-channel
m
( B-channel, n 
R-channel) for several weighting constant . The hori-
zontal axis represents I ( ).
Fig. 5. Denoising performance of B-channel images. The horizontal axis rep-
resents the image index. The green (online version) dotted graph represents the
m
case where G-channel is used as a reference ( B-channel,n G-channel),
and the red (online version) solid graph represents the case where R-channel is
used as a reference ( m B-channel, n R-channel).
that is also related with the spatial complexity of
images.
We also evaluate denoising performance by adopting the
G-channel as a reference ( B-channel, G-channel).
As expected from their spectrum (see Fig. 1), the correlations
between G and B are usually stronger than the correlations
between R and B. Actually, the range of in our dataset
is 1.28–4.13 in the former case, and 1.80–5.76 in the latter
case. Therefore, as shown in Fig. 5, the performance can be
improved (about 2 dB) by adopting G-channel as a reference.
This indicates that a channel showing strong correlations (with
a current channel) should be selected as a reference channel
when there are multiple references. Since the relative strength
of channel correlations can be known a priori, the selection is
not a difficult task. We have also tested a method considering
multiple references simultaneously by assuming
(40)
However, this naive approach does not improve the overall per-
formance, rather, its performance is close to poor one among
two possible choices.
Finally, we analyze the effects of nonideal references. In ex-
periments, we adopt ideal/nonideal G-channel images as ref-
607
Fig. 6. Denoising performance of B-channel images when the G-channel is
used as a reference. The horizontal axis represents the image index sorted ac-
cording to the performance of the single-channel method. The red (online ver-
sion ) dashed line (the lowest) represents the denoising performance of a single-
channel algorithm (BLS-GSM) [23]. Other solid lines represent the performance
of our method using ideal/nonideal reference images. See text for more details.
erences and perform the denoising of B-channel whose noise
level is ( B-channel, G-channel). Precisely,
we adopt the greedy method presented in Section II-C in order
to handle non-ideal references: we denoise a reference image
using a single-channel algorithm (BLS-GSM) and apply the
proposed method by assuming the denoised image is a true one,
i.e., we have optimized (13) [which is equivalent to (39)]. Ex-
perimental results are summarized in Fig. 6. The red (online ver-
sion) dashed line (the lowest one) represents the denoising per-
formance of a single-channel algorithm [23]. Other solid lines
represent the performance of our method: the top line represents
the case in which the ideal reference channel is used, and the
other three lines represent the cases where noise variance of the
reference channel is , and respectively. As can be
seen in Fig. 6, our method significantly improves the denoising
performance even if references are corrupted. However, as ex-
pected, the performance degrades as the noise level of a refer-
ence is increasing.
B. Satellite Image Fusion (Image Interpolation)
One of the most important issues in satellite image processing
is to provide a composite by merging the images from several
sensors [20]. It is because some sensors of a satellite provide
images that are meant to identify objects spatially but not spec-
trally (PAN images), and other sensors to identify spectrally
but not spatially (spectral images such as R, G, B, and NIR
images). For example, IKONOS produces 1-m high-resolution
PAN images and 4-m low-resolution spectral images. Therefore,
they are fused into an image so that we can identify natural
and manmade objects from the result. For this goal, low-res-
olution images (spectral images) are expanded to higher reso-
lution ones with the aid of high-resolution PAN images. Most
widely used image fusion algorithms are Intensity-Hue-Satu-
ration (IHS) method [28] and AWT method [29], [30]. These
methods provide high-resolution images by adding high-fre-
quency of PAN images to the spectral ones.
In the proposed image fusion method, corresponds to a
spectral image to be expanded, and is a PAN image as il-
lustrated in Fig. 7. Unlike the case in the previous subsection,
608 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 3, MARCH 2011

2
Fig. 7. Illustration of the proposed fusion process. The resolution of I is 4
2
higher than that of K . For example, the size of K is 128 128 and the size
2
of I is 512 512.

the resolution of (multispectral image) is smaller than the

resolution of its reference image. Therefore, we use the obser-
vation model in [3] for the first term in (36), which assumes that
low resolution images are obtained through low-pass filtering
and down-sampling of high resolution images. Under these as-
sumptions, in (36) is given by

(41) Fig. 8. One of image sets used in the image fusion experiments. (a) PAN image
2 2 2
(512 512). (b) R image (128 128). (c) G image (128 128). (d) B image
2
(128 128).
where is the variance of observation noise, is a raster-scan
ordered pixel vector of the low-resolution image is from
high-resolution image by sequentially placing each of 4 4 TABLE II
PSNR COMPARISON OF IMAGE FUSION
blocks, and is a filtering and decimation matrix:

..
.

(42)

where is scaling factor (in this case, , see [3] for more
details). Note that we can use this model because images are
taken at the same viewpoint with the same shot angle as shown
in Fig. 8. Otherwise, multisensor image registration methods
should be applied [31].
We can get the result by minimizing the cost function using
the conjugate gradient algorithm [27]
(43)
where the first term encodes the condition that the interpolated
result should be consistent with the observation. On the other
hand, the second term in (43) prevents the structure misalign-
ment between the interpolated one and the reference . Since
becomes small only when has sim-
ilar structure to (as illustrated in Fig. 3), the minimization of
(43) naturally results in which is consistent with the refer-
ence image. Note that, in our method, interpolation methods are
not explicitly used. Rather, the overall process (an interpolation
process considering a reference image as well as an observed
image) is modeled as a single optimization problem.
For the objective evaluation of our method, we measure
the PSNR of enlarged images (one of image sets is shown in
Fig. 8) and summarize the result in Table II. As can be seen,
our method yields more than 3-dB higher PSNR compared with
the IHS method and 1-dB higher PSNR compared with the
AWT method. Moreover, the proposed method suppresses color
distortion which is the major drawback of previous methods.
Figs. 9 and 10 show that the proposed method introduces
relatively small amount of color distortion, and does not suffer
from ringing artifacts which are observed in AWT method [e.g.,
Fig. 9(c)].
C. Chrominance Image Interpolation With Denoising
Chrominance image processing has received relatively less
attention and was independently treated in most applications
[22]. However, by considering the luminance images having
higher SNR and/or higher resolution in the processing of
chrominance signals, the performance is expected to be im-
proved. A typical example may be the conventional image/video
standards with 4:2:0 format: the chrominance channel images
(e.g., Cb and Cr channels) have low resolution and low SNR
compared with luminance channel images. Hence, we can
apply the proposed framework to chrominance image denoising
and interpolation (note that our method cannot be applied to the
KOO AND CHO: DESIGN OF INTERCHANNEL MRF MODEL FOR PROBABILISTIC MULTICHANNEL IMAGE PROCESSING 609

Fig. 9. Subjective comparison of satellite image fusion. (a) Original image.

(b) IHS method. (c) AWT method. (d) proposed method.
Fig. 11. Denoising and interpolation experiments on chrominance channel.
(a) Original Cb channel of Lena. (b) Corrupted Cb (PSNR = 25:0 dB).
(c) BLS-GSM + bicubic interpolation (PSNR = 35:69 dB). (d) Proposed
method (PSNR = 36:62 dB).

Fig. 10. Subjective comparison of satellite image fusion. (a) Original image.
(b) IHS method. (c) AWT method. (d) Proposed method.

standards that treat RGB components exactly in the same way,
e.g., RGB 4:4:4 format [32]). In this application, means a
chrominance channel image and means a luminance channel
image. The cost function is similar to that of the satellite
image fusion application because this application also uses the
observation model in [3] . luminance image by using BLS-GSM algorithm [23], and then,
Since a noise-free reference channel is not available (only
its noisy observation is available in practice), we also adopt the
greedy method presented in Section II-C: we first denoise the
Fig. 12. Denoising and interpolation experiments on chrominance channel:
(a) Original Cb channel of Pepper. (b) Corrupted Cb (PSNR = 25:0 dB).
(c) BLS-GSM + bicubic interpolation PSNR = 35.03 dB. (d) Proposed
method (PSNR = 35.97 dB).
our method is applied. The proposed method is compared with
the conventional channel independent bicubic interpolation for
Lena and Pepper images as shown in Figs. 11 and 12. For fair
610
Fig. 13. Cropping and magnification of Fig. 12. (a) BLS-GSM with bicubic
interpolation. (b) Proposed method. (c) Original image.
comparison, the BLS-GSM algorithm is applied to each channel
in this case. As can be seen in the figures, the proposed method
shows about 1-dB PSNR gains. Besides the PSNR gains, our
method provides better subjective quality as can be observed in
Fig. 13.
VI. CONCLUSION
In the conventional MRF-based image processing algorithms,
rich information between the channels has not been taken into
consideration. Hence, we have developed an approach that ex-
ploits the channel correlations in terms of the conditional prob-
ability. Our method provides a principled approach to exploit
the information of other channels, for the processing of a given
channel. The proposed algorithm has been applied to several
applications and experimental results show that the proposed
method improves the performance subjectively and objectively.
REFERENCES
[1] S. Geman and D. Geman, “Stochastic relaxation, gibbs distributions,
and the bayesian restoration of images,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. PAMI-6, no. 7, pp. 721–741, Jul. 1984.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 3, MARCH 2011
[2] D. Geman and G. Reynoids, “Constrained restoration and the recover
of discontinuities,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 14,
pp. 367–383, 1992.
[3] R. R. Schultz and R. L. Stevenson, “A bayesian approach to image
expansion for improved definition,” IEEE Trans. Image Process., vol.
3, no. 3, pp. 233–242, May 1994.
[4] S. Zhu, Y. Wu, and D. Mumford, “Filters, random fields and maximum
entropy (frame): Towards a unified theory for texture modeling,” Int. J.
Comput. Vis., vol. 27, no. 2, pp. 107–126, 1998.
[5] A. Levin and Y. Weiss, “User assisted separation of reflections from a
single image using a sparsity prior,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 29, no. 9, pp. 1647–1654, Sep. 2007.
[6] A. Levin, R. Fergus, F. Durand, and B. Freeman, “Image and depth
from a conventional camera with a coded aperture,” in Proc. SIG-
GRAPH, 2007.
[7] S. Zhu and D. Mumford, “Prior learning and gibbs reaction-diffusion,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 11, pp. 1236–1250,
Sep. 1997.
[8] S. Roth and M. Black, “Fields of experts: A framework for learning
image priors,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognition,
2005.
[9] X. Lan, S. Roth, D. Huttenlocher, and M. J. Black, “Efficient belief
propagation with learned higher-order Markov random fields,” in Proc.
Eur. Conf. Comput. Vis., 2006, pp. 269–282.
[10] Y. Weiss and W. T. Freeman, “What makes a good model of natural
images,” in Proc. IEEE Conf. Comput. Vis.Pattern Recognition, 2007.
[11] R. Kimmel, “Demosaicing: Image reconstruction from color ccd sam-
ples,” IEEE Trans. Image Process., vol. 8, no. 9, pp. 1221–1228, Sep.
1999.
[12] X. Li and M. Orchard, “New edge-directed interpolation,” IEEE Trans.
Image Process., vol. 10, no. 10, pp. 1521–1527, Oct. 2001.
[13] D. D. Muresan and T. W. Parks, “Demosaicing using optimal re-
covery,” IEEE Trans. Image Process., vol. 14, no. 2, pp. 267–278, Feb.
2005.
[14] M. Bartkowiak, “Improved interpolation of 4:2:0 colour images to
4:4:4 format exploiting inter-component correlation,” in Proc. Eur.
Signal Process. Conf., 2004.
[15] A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimiza-
tion,” in Proc. SIGGRAPH, 2004, pp. 689–694.
[16] A. Zomet and S. Peleg, “Multi-sensor super resolution,” in Proc. IEEE
Workshop Appl. Comput. Vis., 2002, pp. 27–31.
[17] S. Z. Li, Markov Random Field Modeling in Image Analysis. New
York: Springer-Verlag, 2001.
[18] L. Goffman-Vinopal and M. Porat, “Color image compression using
inter-color correlation,” in Proc. Int. Conf. Image Process., 2002, vol.
2, pp. 353–356.
[19] B. C. Song, Y. G. Lee, and N. H. Kim, “Block adaptive inter-color
compensation algorithm for RGB 4:4:4 video coding,” IEEE Trans.
Circuits Syst. Video Technol., vol. 18, no. 10, pp. 1447–1451, Oct. 2008.
[20] Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparitive
analysis of image fusion methods,” IEEE Trans. Geosci. Remote Sens.,
Jun. 2005.
[21] E. Bennett, J. Mason, and L. McMillan, “Multispectral bilateral video
fusion,” IEEE Trans. Image Process., vol. 16, no. 5, pp. 1185–1194,
May 2007.
[22] C. Fogg, D. J. LeGall, J. L. Mitchell, and W. B. Pennebaker, MPEG
Video Compression Standard. : Springer, 1996.
[23] J. Portialla, V. Sterla, M. J. Wainwright, and E. P. Simoncelli, “Image
denoising using scale mixture of Gaussians in the wavelet domain,”
IEEE Trans. Image Process., vol. 12, no. 11, pp. 1338–1351, Nov.
2003.
[24] D. Muresan and T. Parks, “Adaptively quadratic (aqua) image interpo-
lation,” IEEE Trans. Image Process., vol. 13, no. 5, pp. 690–698, May
2004.
[25] V. Kolmogorov, Y. Boykov, and C. Rother, “Applications of parametric
maxflow in computer vision,” in Proc. IEEE Int. Conf. Comput. Vis.,
Oct. 2007, pp. 1–8.
[26] A. Blake, C. Rother, M. Brown, P. Pérez, and P. H. S. Torr, “Interac-
tive image segmentation using an adaptive GMMRF model,” in Proc.
ECCV, 2004, pp. 428–441.
[27] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Nu-
merical Recipes in C: The Art of Scientific Computing. Cambridge,
U.K.: Cambridge Univ. Press, 1992.
[28] M. J. Choi, “A new intensity-hue-saturation fusion approach to image
fusion with a tradeoff parameter,” IEEE Trans. Geosci. Remote Sens.,
vol. 44, no. 6, pp. 1672–1682, Jun. 2006.
KOO AND CHO: DESIGN OF INTERCHANNEL MRF MODEL FOR PROBABILISTIC MULTICHANNEL IMAGE PROCESSING 611

[29] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, “Context-driven Nam Ik Cho (S’86–M’92) received the B.S., M.S.,
fusion of high spatial and spectral resolution images based on oversam- and Ph.D. degrees in control and instrumentation
pled multi-resolution analysis,” IEEE Trans. Geosci. Remote Sens., vol. engineering from Seoul National University, Seoul,
40, no. 10, pp. 2300–2312, Oct. 2002. Korea, in 1986, 1988, and 1992, respectively.
[30] T. Ranchin, B. Aiazzi, L. Alparone, S. Baronti, and L. Wald, “Image fu- From 1991 to 1993, he was a Research Associate
sion-the arsis concept and some successful implementation schemes,” with the Engineering Research Center for Advanced
ISPRS J. Photogramm. Remote Sens., vol. 58, pp. 4–18, 2003. Control and Instrumentation, Seoul National Univer-
[31] M. Irani and P. Anandan, “Robust multi-sensor image alignment,” in sity, Seoul, Korea. From 1994 to 1998, he was with
Proc. Int. Conf. Comput. Vis., Jan. 1998, pp. 959–966. the University of Seoul, Seoul, Korea, as an Assis-
[32] B. C. Song, Y. G. Lee, and N. H. Kim, “Block adaptive inter-color tant Professor of Electrical Engineering. He joined
compensation algorithm for RGB 4:4:4 video coding,” IEEE Trans. the School of Electrical Engineering, Seoul National
Circuit Syst. Video Technol., vol. 18, no. 10, pp. 1447–1451, Oct. 2008. University, in 1999, where he is currently a Professor. His research interests in-
clude image processing and adaptive filtering.
Hyung Il Koo (S’09–M’10) received the B.S., M.S.,
and Ph.D. degrees in electrical engineering and com-
puter science from Seoul National University, Seoul,
Korea, in 2002, 2004, and 2010, respectively.
Currently, he is a Senior Research Engineer with
Qualcomm R&D Center, Seoul, Korea, where he
is responsible for developing core technology in
computer vision. His research interests include
image processing, pattern recognition, and computer
vision.