EE6310, Image and Video Processing

Linear Image Filtering

February 16, 2024

Linear Image Filtering

I Wraparound and Linear Convolution

I Linear Image Filters
I Linear Image Denoising
I Linear Image Restoration
I Filter Banks
Linear Image Filtering – Illusions

Figure: Who is angry?

Linear Image Filtering – Illusions

Figure: Close or far away?

Linear Image Filtering – Circular Convolution

I Modifying DFT of image changes its appearance - for e.g.,

applying a zero-one mask
I What happens if two image DFTs are multiplied pointwise?
N
J˜ = I˜1 I˜2
I A very important question whose answer has profound
consequences in image processing
I The inverse DFT of J˜ is:
1 P P ˜
N 1M 1
J(i, j) = NM J(u, v )WN ui WN vj
u=0 v =0
Linear Image Filtering – Circular Convolution
N
I Replacing J(u,
˜ v ) with I˜1 (u, v ) I˜2 (u, v )

1 P P ˜ N
N 1M 1
J(i, j) = NM I1 (u, v ) I˜2 (u, v )WN ui WN vj
u=0 v =0
=
1 P P
N 1 M 1 NP1 MP1 NP1 MP1
NM { I1 (n, m)WNun WM vm }{ I2 (p, q)WNup WM
vq
}WN u
u=0 v =0 n=0 m=0 p=0 q=0
=
NP1 MP1 NP1 MP1 NP1 MP1
1 u(n+p i) v (m+q j)
NM I1 (n, m) I2 (p, q) WN WM
n=0 m=0 p=0 q=0 u=0 v =0
NP1 MP1 NP1 MP1
1
= NM I1 (n, m) I2 (p, q).NM. (n + p i, m + q j)
n=0 m=0 p=0 q=0
NP1 MP1
= I1 (n, m)I2 [(i n)N , (j m)M ]
n=0 m=0
NP1 MP1
= I1 [(i p)N , (j q)M ]I2 (p, q) = I1 #I2
p=0 q=0
Linear Image Filtering – Circular Convolution

I pN = p mod N
I I1 #I2 is the circular convolution of I1 with I2
I Like linear convolution except with indices taken modulo N,
M

Figure: Depicting circular convolution

Linear Image Filtering – Circular Convolution

Figure: Without circular convolution

I Modulo arithmetic defines the product for all

0  i  N 1; 0  j  M 1
Linear Image Filtering – Circular Convolution

Figure: Cyclic convolution

I Overlay of periodic extension of I2 for all

0  i  N 1; 0  j  M 1
Linear Image Filtering – Computing Circular Convolution

I Direct computation of I1 #I2 is simple but expensive

I For an N ⇥ M image:
I For each of NM coordinates: NM multiplications and additions
I For the entire image: NM.NM multiplications
I If N = M = 512, it turns out 236 = 6.9 ⇥ 1010
I An alternative is to use FFT to compute #
N
I J = I1 #I2 = IFFTN⇥M [FFTN⇥M [I1 ] FFTN⇥M [I2 ]]
I Computing an N ⇥ M FFT is O[NM.log(NM)], and so is the
computation of #
I To make # useful, it must be modified
Linear Image Filtering – Linear Convolution

I Cyclic convolution is a consequence of the DFT, which is a

sampled DSFT.
I If two DSFTs are multiplied together, then useful linear
convolution results:
J˜D = ID1
˜ ID2
˜ =) J(i, j) = I1 (i, j) ⇤ I2 (i, j)
I Cyclic convolution is an artifact of sampling the DSFT –
which causes spatial periodicity
I Most of circuit theory, optics and analog filter theory based on
linear convolution
Linear Image Filtering – Linear Convolution
I Linear digital filter theory also requires the concept of linear
convolution
I It turns out that circular convolution can be used to
compute linear convolution
I However, circular convolution has drawbacks and cannot be
used as-is
I Consider the average filter: output is the average of pixel
values in a square window

Figure: Averaging filter

Linear Image Filtering – Averaging Filter

Figure: Averaging using circular convolution

The averaging filter may be expressed as the circular convolution

of the image with the image of a square with intensity 1/M where
M is the number of pixels in the square.
Linear Image Filtering – Averaging Filter

Figure: Undesirability of circular convolution

I Wraparound e↵ects occur at image borders

I Desirable to average only neighboring elements and weight
images based on spatial ordering rather than DFT-induced
periodic ordering
I Large filters worsen e↵ect
Linear Image Filtering – Linear Convolution by Zero
Padding

I Conceptually simple: Pad images to be convolved with

zeros
I Typically, both image arrays are doubled in size
I Wraparound eliminated since the “moving” image is
weighted by zero outside image domain

Figure: Zero padded images

Linear Image Filtering – Visualizing Wraparound
Cancellation

Figure: Wraparound cancellation

Remember, summation takes place within blue region

(0  i  2N 1; 0  j  2M 1)
Linear Image Filtering – DFT Computation of Linear
Convolution

I Let J 0 , I10 , I20 be zero-padded 2N ⇥ 2M versions of J = I1 ⇤ I2

I If J 0 = I10 #I20 = IFFT2N⇥2M [FFT2N⇥2M [I10 ]FFT2N⇥2M [I20 ]], the
N ⇥ M image with elements
J(i, j) = J 0 (i, i); 0  i  N 1, 0  j  M 1 contains the
linear convolution result
I Therefore, by multiplying zero-padded DFTs and taking the
IFFT, we get J = I1 ⇤ I2
I The linear convolution is larger than N ⇥ M (2N ⇥ 2M) but
the interesting part is contained in N ⇥ M region of J
I To convolve an N ⇥ M image with a P ⇥ Q filter:
I If P, Q < N, M, pad filter with zeros to size N ⇥ M
I If P, Q << N, M, faster to convolve in space domain
Linear Image Filtering – Direct Linear Convolution

I Assume I1 and I2 are not periodically extended and assume

I1 (i, j) = I2 (i, j) = 0 if i < 0, J < 0 and i > N 1, j > M 1
I In this case,
NP1 MP1
J(i, j) = I1 (i, j) ⇤ I2 (i, j) = I1 (n, m)I2 (i n, j m)
n=0 m=0
Linear Image Filtering – Goals

I Linear filtering refers to a process that transforms an image I

by linear convolution
I Goals include transforming an image:
I Into one of better quality
I With certain features enhanced
I With certain features de-emphasised or removed
I Specifically:
I Smoothing: remove noise due to bit errors, transmission etc
I Deblurring: increase sharpness of blurred images
I Sharpening: emphasize significant features like edges
I Combinations of these
Linear Image Filtering – Characterizing Linear Filters

I Any linear filter can be characterized in one of two equivalent

ways:
I The impulse response H = H(i, j)
I The frequency response H̃ = H̃(u, v )
I H̃ = DFT[H], H = IDFT[H̃]
I The frequency response describes how image frequencies are
a↵ected by the system
p
I Since H̃(u, v ) = |H̃(u, v )|exp{ 1\H̃(u, v )}, an image
frequency component at (u, v ) = (a, b) is amplified or
attenuated by |H̃(a, b)| and shifted by \H̃(a, b)
Linear Image Filtering – Frequency Response Example

I If the input to a system H is a cosine image:

cj
I (i, j) = cos[2⇡( Nb i + Mc
j)] = 12 {WNbi WM + WN bi WM cj }
I The output is:
J(i, j) = I (i, j)⇤H(i, j) = |H̃(b, c)|cos[2⇡( Nb i + M
c
j)+\H̃(b, c)]
I As with 1-D signals, the impulse response is an e↵ective way
to model responses since:
NP1 MP1
I (i, j) = I (n, m) (i n, j m)
n=0 m=0
Linear Image Filtering – Linear Filter Design

I Given analog spec: Hc (x, y ) ()

F
H̃c (⌦, ⇤)
I Two simple methods of filter design:
I Space-sampled approximation:
(X = 1, Y = 1), H(i, j) = HC (i, j); 1 < i, j < 1
I Frequency-sampled approximation using DFT:
P
1 P
1
H̃(u, v ) = H̃c [( Nu n), ( M
v
m)]
n= 1 m= 1
Linear Image Filtering – Space-Sampled Approximation

I Truncate the continuous response:

Htrunc (i, j) = HC (i, j); 0  |i|  (N/2) 1; 0  |j|  (M/2) 1
I Truncation of the analog spec leads to Gibbs phenomena at
jump discontinuities of H̃C (⌦, ⇤)
IDFT
I DFT/IDFT pair H̃trunc (u, v ) () Htrunc (i, j)
Linear Image Filtering – Frequency Sampled Approximation

I H̃fs (u, v ) = H̃C ( Nu , M

v
); 0  |u|  (N/2) 1, 0  |v | 
(M/2) 1
I DSFT NOT specified between samples
I CFT is centered and non-periodic
DFT
I The discrete impulse response is then: Hfs (i, j) () H̃fs (u, v )
Linear Image Filtering – Low-Pass, Band-Pass and
High-Pass Filters

I Low-pass: Attenuates all but “lower” frequencies

I smooth noise
I blur detail to highlight gross features
I Band-pass: Attenuates all but “middle” frequencies
I Special-purpose
I High-pass: Attenuates all but “higher” frequencies
I enhance image detail and contrast
I remove image blur
Linear Image Filtering – Example Low-Pass Filter

I To smooth an image: replace each pixel in a noisy image by

the average of its M ⇥ M neighbors
I How to pick window size M?
I Example: for a 512⇥512 image, typical window sizes range
from 3⇥3, 5⇥5, . . ., 15⇥15 (lots of smoothing)
Linear Image Filtering – Example Low-Pass Filter

I Design an ideal
( low-pass filter in the DFT domain:
p
1, if u + v 2  Ucuto↵
2
H̃(u, v ) =
0, otherwise
I Possibly useful if radial frequency Ucuto↵ can be estimated in
the original image

Figure: Ideal low-pass filter

Linear Image Filtering – Example Low-Pass Filter

I The gaussian filter with frequency response

H̃C (⌦, ⇤) = exp[ 2(⇡ )2 (⌦2 + ⇤2 )], hence
H̃(u, v ) = exp{ 2⇡ 2 2 [( Nu )2 + ( M
v 2
) ]}
I Response falls quickly at high frequencies
I Important low-pass filter
Linear Image Filtering – Example Band-Pass Filter

I Define a BP filter as the di↵erence of two LPFs that di↵er

only by a scaling factor
I Common example is a di↵erence of gaussian (DOG) filter:
H̃C (⌦, ⇤) = exp[ 2(⇡ )2 (⌦2 +⇤2 )] exp[ 2(K ⇡ )2 (⌦2 +⇤2 )]
hence H̃(u, v ) =
exp{ 2⇡ 2 2 [( Nu )2 + ( M
v 2 v 2
) ]} exp{ 2K 2 ⇡ 2 2 [( Nu )2 + ( M ) ]}
I Typically, K ⇡ 1.5
I DOG filters are very useful for image analysis and human
visual modeling
Linear Image Filtering – Example High-Pass Filter

I Laplacian filter is important as well

I A severely truncated approximation is
H̃C (⌦, ⇤) = A(⌦2 + ⇤2 ), hence H̃(u, v ) = A[( Nu )2 + ( M
v 2
) ]
I An approximation to the Fourier transform of the continuous
@2 @2
Laplacian: r2 = @x 2 + @y 2
Linear Image Filtering – Linear Image Denoising

I Linear image denoising is a process of smoothing noise

without destroying image information
I Noise modeled as additive or multiplicative
I Consider additive noise now
I Multiplicative noise better handled by non-linear filtering
Linear Image Filtering – Additive White Noise Model

I Model additive white noise as an image N with random

elements
I Can be thermal noise, channel noise, sensor noise etc.
I Noise may a↵ect the continuous image before sampling:
JC (x, y ) = IC (x, y ) + N(x, y )
Linear Image Filtering – Additive White Noise Model

I Noise is zero-mean if limit of the average of P realizations of

the noise image vanishes as P ! 1:
1 P
P
P NC ,p (x, y ) ! 0 for all (x, y ) as P ! 1
p=1
I On average, noise falls close to 0
Linear Image Filtering – Spectrum of White Noise

I The noise energy spectrum is NC (⌦, ⇤) = F{Nc (x, y )}

I If noise is white, then, on average, the energy spectrum
will be flat (or ‘white’):
1 P
P
P |NC ,p (⌦, ⇤)| ! ⌘ for all (⌦, ⇤) as P ! 1
p=1
I ⌘2 is called noise power
Linear Image Filtering – Linear Filtering

I Objective: Remove as much of the high frequency noise as

possible while preserving as much image spectrum as possible
I Achieved by a LPF with large bandwidth (images are fairly
wideband)
Linear Image Filtering – Digital White Noise

I Similar model for digital zero-mean additive white noise:

J = I + N
observed original noise
I On average, elements of N will be zero
I DFT of noisy image is: J˜ = I˜ + Ñ
observed original noise
I On average, the noise DFT will contain a broad band of
frequencies
Linear Image Filtering – Denoising with Average Filter

I To smooth an image: replace each pixel in a noisy image by

the average of its M ⇥ M neighbors
I Rationale: Averaging elements reduces the noise mean
towards zero
I How to pick window size M? A tradeo↵ between noise
smoothing and image smoothing
I Example: for a 512⇥512 image, typical window sizes range
from 3⇥3, 5⇥5, . . ., 15⇥15 (lots of smoothing)
I Linear filtering the image with zero padding a↵ects the
image and noise spectra in the same way: N N
K = H ⇤ J = H ⇤ I + H ⇤ N =) K̃ = H̃ I˜ + H̃ Ñ
Linear Image Filtering – Denoising with Ideal Low-Pass
Filter

I Design an ideal
( low-pass filter in the DFT domain:
p
1, if u + v 2  Ucuto↵
2
H̃(u, v ) =
0, otherwise
I Possibly useful if radial frequency Ucuto↵ can be estimated in
the original image

Figure: Ideal low-pass filter

Linear Image Filtering – Denoising with Gaussian Filter

I The isotropic Gaussian filter is an e↵ective smoother:

2 2
H̃(u, v ) = exp[ 2⇡ 2 2 ( u N+v
2 )]

I More weight given to “closer” neighbors

I DFT design: set half-peak bandwidth to Ucuto↵ and solve for
U2
p p
: exp[ 2⇡ 2 2 ( cuto↵
N2
)] = 12 =) = ⇡UNcuto↵
log 2 ⇡
N
0.19( Ucuto↵ )
Linear Image Filtering – Summary of Denoising Filters

I Average filter: Noise leakage through frequency ripple

(spatial discontinuity)
I Ideal low-pass filter: Ringing from spatial ripple (frequency
discontinuity)
I Gaussian filter: No ringing, no discontinuities and no leakage
Linear Image Filtering – Minimum Uncertainty

I Amongst all real functions and in any dimension, the

no-ripple Gaussian functions
R uniquely
R minimize the
|xf (x)|2 dx |uf (u)|2 du
uncertainty principle: ( |f (x)|2 dx )( |f (u)|2 du ) 14
R R

I Similar for y , v
I Minimal simultaneous space-time duration
Linear Image Filtering – Linear Image Deblurring

I Often a digital image is corrupted by a linear process

I Motion blur, defocusing etc. causes of blurring
I Blurring can be modeled using linear convolution:
JC (x, y ) = GC (x, y ) ⇤ IC (x, y )
=) J˜C (⌦, ⇤) = G˜C (⌦, ⇤)I˜C (⌦, ⇤)
Linear Image Filtering – Digital Blur Function

I The sample image will be of the form: J = G ⇤ I and

N
J˜ = G̃ I˜
I The distortion G is almost always low-pass
I Goal: Use digital filtering to reduce blur - a very hard
problem!
Linear Image Filtering – Inverse Filter Deblurring

I Often possible to estimate G

I Physics of the situation helps:
I Motion blur along one direction. Determining this direction
can help with filter design
I Modulation transfer function (MTF) of camera system can be
determined and a suitable digital deblurring filter can be
designed
Linear Image Filtering – Deconvolution

I Reversing the linear blur G is deconvolution. It is the

1
inverse filter of the distortion: G̃inverse (u, v ) = G̃ (u,v )
if
G̃ (u, v ) 6= 0 for any (u, v )
N N
I The restored image is then: K̃ = G̃inverse G̃ I˜ = I˜!
Linear Image Filtering – Blur Estimation

I An estimate of blur G might be obtainable

I The inverse of low-pass blur is high-pass
I Designer must be careful at high frequencies

Figure: Inverting blurring filter

Linear Image Filtering – Blur Estimation

I Real world far from “ideal”

I Blur frequency response become zero
I Directly inverting G̃ (u, v ) meaningless when G̃ (u, v ) = 0
I Frequencies zeroed by linear distortion are irrecoverable
I The best one can do is to reverse distortion at the non-zero
values
Linear Image Filtering – Pseudo-Inverse Filter

I The pseudo-inverse ( filter is defined as:

1/G̃ (u, v ), if G̃ (u, v ) 6= 0
G̃p inverse (u, v ) =
0, otherwise
I No attempt to recover lost frequencies
I Psuedo-inverse set to 0 in regions of missing frequencies
I Spurious noise frequencies eradicated
Linear Image Filtering – Blur and Additive Noise

I Things get worse when I is distorted by linear blur G and

additive noise N: J = G ⇤ I + N
I An example is image sent over a noisy communication channel
N˜
I The DFT: J˜ = G̃ I + Ñ
Linear Image Filtering – Failure of Inverse Filters

I Filtering with a linear filter H will produce:

K =HN ⇤ J =N H ⇤G ⇤N I + H ⇤ N or
K̃ = H̃ G̃ ˜
I + H̃ Ñ
I We have a problem:
I Low-pass filter smoothes over noise but doesn’t remove blur
I High-pass filter amplifies noise
Linear Image Filtering – Wiener Filter

I Wiener filter (after Norbert Wiener) is a minimum mean

squared error (MMSE) filter
I For blur G and white noise N, G̃Wiener (u, v ) = G̃ (u,v )
|G̃ (u,v )|2 +⌘ 2
I Noise factor ⌘ usually not known - commonly estimated from
flat image regions
Linear Image Filtering – Wiener Filter Rationale

I In case of no noise (⌘ = 0), the filter reduces to

G̃ (u,v ) 1
G̃Wiener (u, v ) = |G̃ (u,v )|2
= G̃ (u,v )
I This is the highly desirable inverse filter
I In case of no blur, the filter reduces to G̃Wiener (u, v ) = 1
1+⌘ 2
I This essentially scales the variance to minimize MSE
Linear Image Filtering – Pseudo-Wiener Filter

I To handle zeroed8frequencies, define a pseudo-Wiener filter:

< G̃ (u,v ) , if G̃ (u, v ) 6= 0
G̃Wiener (u, v ) = |G̃ (u,v )|2 +⌘2
:0, otherwise
I Noise in the zeroed frequency region eradicated