Prof. A.

Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

LECTURE 9:
IMAGE DE-NOISING
• How do we separate signal from noise?
– Image models
– Gaussian vs. Impulsive Noise
• Linear Shift-Invariant Filtering
• Local Adaptive Filtering
– Local LMMSE Filter
• Non-linear Filtering
– Median and Order-Statistics Filtering
– Wavelet Shrinkage
– Bilateral Filtering
• Non-Local Filtering
– NLM Filtering
– BM3D Filtering
Chapter 3 Image Filtering 1
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

How Do We Separate Signal from Noise?

• Our ability to separate the signal from noise depends on how
well various models allow such separation.
• Noise Models:
– White
– Signal-independent
• Image Models:
– Local smoothness, low-pass in the Fourier domain
– Sparsity in transform domain

Chapter 3 Image Filtering 2

 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Linear Shift-Invariant Filtering

• An LSI noise reduction filter attenuates frequencies where the noise
power exceeds the signal power.
• In LSI denoising, there is an inherent tradeoff between noise reduction
and blurring of image detail because any LSI denoising filter is
essentially a low-pass filter, also called a smoothing filter, which
suppresses high-frequencies.

Sf()

Sv()=v2


Chapter 3 Image Filtering 3
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

LSI Filtering in DFT Domain: Wiener Filter

• The LMMSE estimate ̂ , of the ideal image , is given by

̂ , , ,

where , is noisy image and , is the impulse response of the filter.

• The principle of orthogonality states that estimation error, , ̂ , ,
at each pixel should be orthogonal to every sample of the observed image,
, ̂ , , , , ̂ , , 0
for all , and , . where the inner product ·,· is defined in terms of
the expectation operator · . Thus, orthogonality means uncorrelatedness.
• Substituting and simplifying the resulting expression, we obtain

, ,

, for all , and ,

Chapter 3 Image Filtering 4
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Wiener Filter (cont’d)

• The autocorrelation function of the observations
, , ,
• The cross-correlation between the ideal image and the observed image
, , ,
• The double summation can be expressed as the discrete Wiener-Hopf equation
, ∗∗ , ,
which defines the impulse response of the noncausal, IIR Wiener filter, also
known as the unrealizable Wiener filter.
• This IIR filter is unrealizable because an infinite time delay is required to
compute an output sample.
• We can compute the frequency response of the Wiener filter by taking the 2D
Fourier transform of both sides of the discrete Wiener-Hopf equation to obtain
,
,
,
Chapter 3 Image Filtering 5
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Wiener Filter (cont’d)

• Given the noise model
, , ,
, , , , ,
and
• , , , , ,
, ,
where we assume that the image and noise are uncorrelated.
• The power spectra , , and , , , are
the 2D Fourier transform of the correlation functions.
• Then, the frequency response of the Wiener filter becomes
,
,
, ,
• We observe that , since the noise is white and the power spectrum of
original image can be estimated from the noisy image as , , .
Chapter 3 Image Filtering 6
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Wiener Filter: DFT Domain Implementation

• A realizable approximation to the filter can be obtained by frequency-sampling
design, where the filter frequency response , is sampled in the frequency
domain using samples. This filter can be efficiently implemented using
fast Fourier transform (FFT).
• The frequency sampling design is equivalent to approximating the impulse
response , of the IIR filter with an FIR filter with the impulse
response , , given by

, ,

• Note that the frequency-sampling design method suffers from spatial-domain

aliasing. In practice, this may be negligible, provided that and are
reasonably large, e.g., 512 or larger.

Chapter 3 Image Filtering 7

 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

FIR LMMSE Filtering

• The FIR LMMSE filter can be expressed in vector-matrix form as

where and are 1 vectors formed by the estimated and observed pixels, and
is an matrix formed by coefficients of the FIR filter impulse response.
• The principle of orthogonality can be stated in vector-matrix form as
zero matrix
which states that every element of is uncorrelated with every element of y.
Then, , which can be simplified as

• Hence, the FIR LMMSE filter operator H can be obtained as

where is the auto-correlation matrix of the observed image, and is the
cross-correlation matrix of the ideal and observed images.
Chapter 3 Image Filtering 8
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

FIR Wiener Filter (cont’d)

• Given the observation model, we can easily show, as in the derivation of the IIR
filter, that , and , where and are the auto-
correlation matrices of the ideal image and the observation noise, respectively.
• Then the filter operator becomes

• Observe that the implementation of the filter requires inversion of a
matrix. For a typical digital image 512 or larger, this is a formidable task.
• Assuming that the image and noise are wide-sense stationary; that is, they have
constant mean (taken as zero without loss of generality) and spatially invariant
correlation matrices, the matrices and are block-Toeplitz. It is common
to approximate block-Toeplitz matrices by block-circulant ones, which can be
diagonalized through the 2D DFT operation. It can readily be shown that the
resulting frequency-domain FIR filter expression is identical to that obtained by
sampling the frequencies , in the IIR Wiener filter expression.
Chapter 3 Image Filtering 9
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Adaptive FIR LMMSE Filter

• We define residual image as
, , ,
where , is a spatially varying mean image. The residual image , is
modeled by white noise; that is, , , , is diagonal. It
follows that the residual of the observed image, , , , ,
where , is local mean, is zero-mean and white, and , , .
• Applying the FIR LMMSE filter to the residual observation vector, ,

• Since is diagonal, the vector-matrix expression simplifies to the scalar form
,
̂ , , , ,
,
with a predictor-corrector structure and , ⁄ , , is the
filter gain.

Chapter 3 Image Filtering 10

 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Adaptive FIR Wiener Filter (cont’d)

• We estimate
1
, , ,
, ∈ ,

• since the noise has zero mean, and

1
, , ,
, ∈ ,

• In order to avoid a negative variance estimate, we have

, max , , 0
• When is small, indicating a uniform image region, the filter gain is is
negligible, and the adaptive LMMSE filter approaches a direct averaging filter.
• When is large compared to , which indicates presence of edges or high
contrast texture, the filter gain approaches one, and edges/texture are preserved
by turning the filter off. Consequently, some noise is left around edges.
Chapter 3 Image Filtering 11
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Local Edge-Adaptive Filtering

• Five filter kernels: 4 edge directions and a non-edge filter

• Edge-direction detection or kernel selection

• Cascade: , where Ti is the local LMMSE filter
applied over the respective kernel

Chapter 3 Image Filtering 12

 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Non-Linear Filtering: Median Filtering

• Median Filtering
̂ , Med , for , ∈ ,

• Weighted Median Filtering

̂ , Med , ◊ , for , ∈ ,

Chapter 3 Image Filtering 13

 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Order-Statistics Filtering
• Order statistics filters require rank ordering (sorting) pixel values in a
neighborhood of the current pixel.
• Alpha-trimmed mean filter is an order statistics filter that combines rank
ordering and averaging. In particular, smallest and largest pixel values
in a local neighborhood are eliminated, and the average of remaining
middle ranked samples are computed as
1
̂ ,
2
• The alpha-trimmed mean filter becomes a mean filter for 0, and it
approaches the median filter as  /2. Therefore, it can be effective
in suppressing a combination of Gaussian noise and impulsive salt-and-
pepper noise by proper selection of .
Chapter 3 Image Filtering 14
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Wavelet Shrinkage Filtering

• Natural images can be represented by a sparse vector in some
transform domain, e.g., an orthogonal wavelet transform domain.
Under sparse image modeling, additive noise leads to very low
SNR on many transform coefficients with small magnitudes.
• All wavelet shrinkage methods consist of the following steps:
– linear forward wavelet transform,
– nonlinear shrinkage (hard or soft thresholding),
– inverse wavelet transform.
Many different wavelet shrinkage methods vary in the details of
wavelet transform implementation and choice of shrinkage function
and threshold value.
Chapter 3 Image Filtering 15
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Wavelet Shrinkage (cont’d)

• For denoising, nearly symmetric orthogonal wavelets are preferred, since orthogonal
transform of white noise is white, and orthogonal transforms preserve mean square error.
• Selection of wavelet analysis and synthesis filters, the number of resolution levels, and
image boundary handling all affect denoising performance. Common choices are 3 or 4
resolution levels using symlet8 wavelets and periodic or symmetric boundary extension.
• Hard thresholding of wavelet coefficient is defined by
if

0 otherwise
• Soft thresholding is defined by
sgn if

0 otherwise
• They may use a universal threshold or adaptive thresholds for different resolution
levels or subbands.
• Several different criteria exist to estimate the best threshold, such as Stein’s
unbiased
Chapter 3 risk
Imageestimate 16
Filtering (SURE), minimax, and Bayesian criteria [Fod 03, Lui 07].
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Bilateral Filtering
• Bilateral filtering has been applied to image denoising by proper choice
of the kernel size , domain parameter and range parameter
to match the noise statistics.
• There is tradoff between the amount of noise reduction and blurring of
edges. The larger the parameters, the closer the filter approaches
Gaussian filtering.
• In the presence of salt-and-pepper noise, a median filter may be applied
prior to bilateral filtering.
• Instead of processing the luminance only, range filtering in the CIE-Lab
color space using a perceptual color similarity metric is a natural way of
processing color images, where only perceptually similar colors are
averaged, and perceptually important edges are preserved without color
bleeding and blurring..
Chapter 3 Image Filtering 17
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Non-Local Filtering: NL Means

• In pixel-wise implementation of the NLM filter, given a noisy image ,
the filtered intensity value ̂ , for each pixel , can be computed as a weighted
average of all similar pixels in the image, given by
̂ ∑ ∈ ,
where ∑ w , is a normalizing constant and denotes a 2q 1
2q 1 window centered at pixel to search for similar patches.
,
• The weights w , depend on sum of absolute value of pixel-
by-pixel similarity comparison of 2r 1 2r 1 patches centered at pixels
and , respectively, given by
1
,
2 1
∈
• The patch size is typically 5×5 ( 2), except maybe in very noisy images.
The search window varies between 21×21 ( 10 for moderate noise) and
35×35 ( 17 for large noise) for computational efficiency reasons.
Chapter 3 Image Filtering 18
 Prof. A. Murat Tekalp Digital Video Processing, 2E, Prentice Hall, 2015

Non-Local Filtering: BM3D

• BM3D is a non-local, blockwise filtering, which consists of three steps:
– Formation of groups: Given the current block , similar blocks are
selected and stacked together as a 3D array. Since there are some minor
structural differences between the blocks, collaborative filtering is applied
in the transform domain as opposed to averaging pixels in the same
location over all blocks.
– Colloborative filtering: Similarity within 3D groups of blocks implies that
the resulting transform shall be sparse. The filter consists of 3D
transformation of 3D groups, shrinkage of transform coefficients, and
inverse 3D transformation.
– Aggregation: The filtered 2D blocks are returned to their original positions.
Because these blocks are overlapping, we obtain multiple estimates for
each pixel, which are combined by a weighted averaging procedure to form
the final denoised estimate for each pixel.
Chapter 3 Image Filtering 19