Digital Image Processing(ET8643EL)/

Image Processing(CS 874) /

Image Processing (CSE1816PE31)

Image Restoration and

Reconstruction

2021
 Degraded images

• Camera: translation, shake, out-of-

• What caused the image to focus …
blur? • Environment: scattered and reflected
light
• Device noise: CCD/CMOS sensor and
• Can we improve the image, circuitry
or “undo” the effects? • Quantization noise
6/22/2021 2
 Overview

• Image enhancement: “improve” an image subjectively.

• Image restoration: remove distortion from image in order
to go back to the “original” objective process.

6/22/2021 3
 Image restoration

► started from the 1950s

► application domains
▪ Scientific explorations
▪ Legal investigations
▪ Film making and archival
▪ Image and video (de-)coding
▪ …
▪ Consumer photography

► related problem: image reconstruction in radio

astronomy, radar imaging and tomography
6/22/2021 4
 Image restoration

► Restoration is an objective process that attempts

to recover an image that has been degraded
▪ A priori knowledge of the degradation phenomenon
▪ Restoration techniques generally oriented toward
modeling the degradation
▪ Application of the inverse process to “recover” the
original image
▪ Involves formulating some criterion (criteria) of
“goodness” that is used to measure the desired result

6/22/2021 5
 Image restoration

► Removal of blur by a deblurring function is an

example restoration technique
► We will consider the problem only from where a
degraded digital image is given
▪ Degradation source will not be considered here
► Restoration techniques may be formulated in the
▪ Frequency domain
▪ Spatial domain

6/22/2021 6
 A Model of Image Degradation/Restoration
Process
► Given g(x,y), some knowledge about H, and some
knowledge about the noise term, the objective is to
produce an estimate of the original image.
▪ The more that is known about H and the noise term the closer the
estimate can be
► Various types of restoration filters are used to accomplish
this. Degraded
Image
Estimated
Input Image Image

6/22/2021 7
 Image degradation/restoration process

► If H is a linear, position invariant process, then the

degraded image can be described as the convolution of h
and f with an added noise term:

► h(x, y) is the spatial domain representation of the

degradation function.
► In the frequency domain, the representation is:

► Each term in this expression is the Fourier transform of the

of the corresponding terms in the equation above.

6/22/2021 8
 Noise Models

► Noise is a random variation of brightness in an image.

► The principal sources of noise in digital images arise during
image acquisition and/or transmission

✓ Image acquisition
e.g., light levels, sensor temperature, etc.

✓ Transmission
e.g., lightning or other atmospheric disturbance in wireless
network
6/22/2021 9
 Noise Models (1)

► White noise
▪ The Fourier spectrum of noise is constant

► With the exception of spatially periodic noise, we assume

▪ Noise is independent of spatial coordinates
▪ Noise is uncorrelated with respect to the image itself

6/22/2021 10
 Noise Models (2)

➢ Gaussian noise
Electronic circuit noise, sensor noise due to poor illumination and/or
high temperature

➢ Rayleigh noise
Range imaging

6/22/2021 11
 Range Imaging (1)

➢ Short for High Dynamic Range Imaging. HDRI is an imaging

technique that allows for a greater dynamic range of exposure
than would be obtained through any normal imaging process.

➢ It is now popularly used to refer to the process of tone

mapping* together with bracketed** exposures of normal
digital images, giving the end result a high, often exaggerated
dynamic range

* Tone mapping is a technique used in image processing and computer graphics to

map a set of colours to another; often to approximate the appearance of HDRI in media
with a more limited dynamic range
** bracketing is the general technique of taking several shots of the same subject using
different or the same camera settings

http://en.wikipedia.org/wiki/High_dynamic_range_imaging
http://www.webopedia.com/TERM/H/High_Dynamic_Range_Imaging.html
6/22/2021 12
 Range Imaging (2)

➢ The intention of HDRI is to accurately represent the wide

range of intensity levels found in real scenes ranging from
direct sunlight to shadows

➢ HDRI, also called HDR (High Dynamic Range) is a feature

commonly found in high-end graphics and imaging software

http://en.wikipedia.org/wiki/High_dynamic_range_imaging
http://www.webopedia.com/TERM/H/High_Dynamic_Range_Imaging.html

6/22/2021 13
 Range Imaging: Examples (1)
http://en.wikipedia.org/wiki/High_dynamic_range_imaging

Tower Bridge in
Sacramento,
CA

6/22/2021 14
 Sydney Harbour
Bridge HDRi
Range Imaging: Examples (2) produces greater
detail and fewer
http://en.wikipedia.org/wiki/High_dynamic_range_imaging shadows

6/22/2021 15
6/22/2021 Old Saint Paul’s Wellinton, New Zealand 16
 http://en.wikipedia.org/wiki/Image:HDRI-Example.jpg

6/22/2021 17
 Noise Models (3)

➢ Erlang (gamma) noise: Laser imaging

➢ Exponential noise: Laser imaging

➢ Uniform noise: Least descriptive; Basis for numerous random

number generators

➢ Impulse noise: quick transients,

such as faulty switching

6/22/2021 18
 Gaussian Noise (1)

The PDF of Gaussian random variable, z, is given by

1 − ( z − z ) 2 /2 2
p( z ) = e
2

where, z represents intensity

z is the mean (average) value of z
 is the standard deviation

6/22/2021 19
 Gaussian Noise (2)

The PDF of Gaussian random variable, z, is given by

1 − ( z − z ) 2 /2 2
p( z ) = e
2

▪ 70% of its values will be in the range

(  −  ), ( +  )
▪ 95% of its values will be in the range

(  − 2 ), (  + 2 )
6/22/2021 20
 Rayleigh Noise

The PDF of Rayleigh noise is given by

2 − ( z − a )2 / b
 ( z − a )e for z  a
p( z ) =  b
 0 for z  a

The mean and variance of this density are given by

z = a + b / 4
b(4 −  )
 =
2

4
The basic shape of this PDF is skewed to the right
6/22/2021
– Can be useful in approximating skewed histograms 21
 Erlang (Gamma) Noise

The PDF of Erlang noise is given by

 a b z b −1 − az
 e for z  0
p( z ) =  (b − 1)!
0 for z  a


The mean and variance of this density are given by

z =b/a
 2 = b / a2

6/22/2021 22
 Exponential Noise

The PDF of exponential noise is given by

 ae − az for z  0
p( z ) = 
0 for z  a
The mean and variance of this density are given by
z = 1/ a
 2 = 1/ a 2

6/22/2021 23
 Uniform Noise

The PDF of uniform noise is given by

 1
 for a  z  b
p( z ) =  b − a
 0 otherwise

The mean and variance of this density are given by

z = ( a + b) / 2
 2 = (b − a) 2 /12

6/22/2021 24
 Impulse (Salt-and-Pepper) Noise

The PDF of (bipolar) impulse noise is given by

 Pa for z = a

p( z ) =  Pb for z = b
0
 otherwise

if b  a, gray-level b will appear as a light dot,

while level a will appear like a dark dot.

If either Pa or Pb is zero, the impulse noise is called

unipolar

6/22/2021 25
6/22/2021 26
 Examples of Noise: Original Image

6/22/2021 27
 Examples of Noise: Noisy Images(1)

6/22/2021 28
 Examples of Noise: Noisy Images(2)

6/22/2021 29
 Periodic Noise

► Periodic noise in an image arises typically from electrical or

electromechanical interference during image acquisition.

► It is a type of spatially dependent noise

► Periodic noise can be reduced significantly via frequency

domain filtering

6/22/2021 30
 An Example of Periodic Noise

6/22/2021 31
 Estimation of Noise Parameters

► By inspection
▪ Histogram in spatial domain
▪ Fourier spectrum (e.g. frequency spikes for periodic noise)

► By calibration
▪ If access to the sensor: collect data for a flat environment
▪ If only images: Estimate PDF from small, near-constant image
patches

6/22/2021 32
 Estimation of Noise Parameters

► Noise parameters can often be estimated by

observing the Fourier spectrum of the image
▪ Periodic noise tends to produce frequency spikes
► Parameters of noise PDFs may be known
(partially) from sensor specification
▪ Can still estimate them for a particular imaging setup
▪ One method
► Capture a set of “flat” images from a known setup (i.e. a uniform gray
surface under uniform illumination)
► Study characteristics of resulting image(s) to develop an indicator of
system noise

6/22/2021 33
 Estimation of Noise Parameters

► If only a set of images already generated by a sensor are

available, estimate the PDF function of the noise from
small strips of reasonably constant background intensity.

Consider a subimage denoted by S , and let ps ( zi ), i = 0, 1, ..., L -1,

denote the probability estimates of the intensities of the pixels in S .
The mean and variance of the pixels in S :
L −1
z =  zi ps ( zi )
i =0
L −1
and  2 =  ( zi − z ) 2 ps ( zi )
i =0

6/22/2021 34
 Estimation of Noise Parameters

► The shape of the noise histogram identifies the

closest PDF match
▪ If the shape is Gaussian, then the mean and variance
are all that is needed to construct a model for the noise
(i.e. the mean and the variance completely define the
Gaussian PDF)
▪ If the shape is Rayleigh, then the Rayleigh shape
parameters (a and b) can be calculated using the mean
and variance
▪ If the noise is impulse, then a constant (with the
exception of the noise) area of the image is needed to
6/22/2021
calculate Pa and Pb probabilities for the impulse PDF 35
 Estimation of Noise Parameters
The shape of the histogram identifies the closest PDF match

6/22/2021 36
 Restoration in the Presence of Noise Only
̶ Spatial Filtering

Noise model without degradation

g ( x, y ) = f ( x , y ) +  ( x, y )
and
G (u , v ) = F ( u , v ) + N ( u , v )

6/22/2021 37
 Spatial Filtering: Mean Filters (1)

Let S xy represent the set of coordinates in a rectangle

subimage window of size m  n, centered at ( x, y ).

Arithmetic mean filter

1
f ( x, y ) = 
mn ( s ,t )S xy
g ( s, t )

6/22/2021 38
 Spatial Filtering: Mean Filters (2)

Geometric mean filter

1
  mn
f ( x, y ) =   g ( s , t ) 
 ( s ,t )S xy 

Generally, a geometric mean filter achieves smoothing

comparable to the arithmetic mean filter, but it tends to
lose less image detail in the process

6/22/2021 39
 Spatial Filtering: Mean Filters (3)
Harmonic mean filter
mn
f ( x, y) =
1

( s ,t )S xy g ( s, t )

It works well for salt noise, but fails for pepper noise.
It does well also with other types of noise like Gaussian
noise.

6/22/2021 40
 Spatial Filtering: Mean Filters (4)

Contraharmonic mean filter


( s ,t )S xy
g ( s, t )Q +1
f ( x, y ) =

( s ,t )S xy
g ( s, t )Q

Q is the order of the filter.

It is well suited for reducing the effects of salt-and-

pepper noise. Q>0 for pepper noise and Q<0 for salt
noise.
6/22/2021 41
 Spatial Filtering: Example (1)

6/22/2021 42
 Spatial Filtering: Example (2)

6/22/2021 43
 Spatial Filtering: Example (3)

6/22/2021 44
 Spatial Filtering: Order-Statistic Filters (1)

Median filter
f ( x, y ) = median  g ( s , t )
( s ,t )S xy

Max filter
f ( x, y ) = max  g (s , t )
( s ,t )S xy

Min filter
f ( x, y ) = min  g (s , t )
( s ,t )S xy

6/22/2021 45
6/22/2021 46
6/22/2021 47
 Periodic Noise Reduction by Frequency
Domain Filtering

The basic idea

Periodic noise appears as concentrated bursts of energy

in the Fourier transform, at locations corresponding to
the frequencies of the periodic interference

Approach

A selective filter is used to isolate the noise

6/22/2021 48
 Perspective Plots of Bandreject Filters

• Circularly symmetric filters

• Remove (noise from) a given frequency range

6/22/2021 49
 A Butterworth bandreject
filter of order 4, with the
appropriate radius and
width to enclose
completely the noise
impulses

6/22/2021 50
 Bandpass Filters

6/22/2021 51
 Perspective Plots of Notch Filters
• Reject or pass specific conjugate impulse pairs (around a center frequency)
• They appear in symmetric pairs about the origin

6/22/2021 52
6/22/2021 53
 Several interference components are present, the methods discussed
in the preceding sections are not always acceptable because they
remove much image information
The components tend to have broad skirts that carry information
about the interference pattern and the skirts are not always easily
detectable.
6/22/2021 54
 Optimum Notch Filtering

It minimizes local variances of the restored estimated

f ( x, y )

Procedure for restoration tasks in multiple periodic

interference

Isolate the principal contributions of the interference

pattern

Subtract a variable, weighted portion of the pattern

from the corrupted image
6/22/2021 55
 Optimum Notch Filtering: Step 1

Extract the principal frequency components of

the interference pattern

Place a notch pass filter at the location of each spike.

N (u, v) = H NP (u , v)G (u , v)
 ( x, y ) = −1 H NP (u, v)G (u, v)

6/22/2021 56
 Optimum Notch Filtering: Step 2 (1)

Filtering procedure usually yields only an approximation of the

true pattern. The effect of components not present in the estimate
of  (x, y ) can be minimized instead by subtracting from g ( x, y )
a weighted portion of  (x, y ) to obtain an estimate of f ( x, y ):

f ( x, y ) = g ( x, y ) − w( x, y ) ( x, y )

One approach is to select w( x, y ) so that the variance of the estimate f ( x, y )

is minimized over a specified neighborhood of every point ( x, y).
6/22/2021 57
 Optimum Notch Filtering: Step 2 (2)

The local variance of f ( x, y ):

a b 2
1  f ( x + s , y + t ) − f ( x, y ) 
 ( x, y ) =
2
 
(2a + 1)(2b + 1) s =− a t =− b  

6/22/2021 58
Assume that w(x,y) remains
essentially constant over the
Optimum
neighborhood gives theNotch Filtering: Step (3)
approximation
w(x+s,
The localy+t) = w(x,y)
variance of f ( x, y ):
a b 2
1  f ( x + s , y + t ) − f ( x, y ) 
 ( x, y ) =
2

(2a + 1)(2b + 1) s =− a t =− b  

b  g ( x + s , y + t ) − w( x + s , y + t ) ( x + s , y + s ) 
2
1 a
 
=   
(2a + 1)(2b + 1) s =− a t =− b −  g ( x, y ) − w( x, y ) ( x, y ) 

   

b  g ( x + s , y + t ) − w( x, y ) ( x + s , y + s ) 
2
1 a
 
=   

(2a + 1)(2b + 1) s =− a t =− b − g ( x, y ) − w( x, y ) ( x, y ) 

   
6/22/2021 59
 Optimum Notch Filtering: Step (4)

The local variance of f ( x, y ):

 g ( x + s, y + t ) − w( x, y ) ( x + s, y + s ) 
2
a b
1
 ( x, y ) =
2
 
(2a + 1)(2b + 1) s =− a t =− b −  g ( x, y ) − w( x, y ) ( x, y ) 

   

 2
( x, y )
To minimize  ( x, y ) ,
2
=0
w( x, y )
for w( x, y ), the result is
g ( x, y ) ( x, y ) − g ( x, y ) ( x, y )
w( x, y ) = 2
 ( x, y ) −  ( x, y )
2

6/22/2021 60
 Optimum Notch Filtering: Example

6/22/2021 61
 Optimum Notch Filtering: Example

6/22/2021 62
 Optimum Notch Filtering: Example

6/22/2021 63
 Optimum Notch Filtering: Example

6/22/2021 64
 Linear, Position-Invariant Degradations

g ( x , y ) = H  f ( x , y )  +  ( x, y )

6/22/2021 65
 Linear, Position-Invariant Degradations

H is linear
H  af1 ( x, y ) + bf 2 ( x, y )  = aH  f1 ( x, y )  + bH  f 2 ( x, y ) 
f1 and f 2 are any two input images.

An operator having the input-output relationship

g ( x, y ) = H  f ( x, y )  is said to be position invariant
if
H  f ( x −  , y −  ) = g ( x −  , y −  )
for any f ( x, y ) and any  and  .
6/22/2021 66
 Linear, Position-Invariant Degradations
 
f ( x, y ) =   f ( ,  ) ( x −  , y −  )d d 
− −

Assume for a moment that  ( x, y ) = 0

if H is a linear operator,
g ( x, y ) = H  f ( x, y ) 

= H   f ( ,  ) ( x −  , y −  ) d d  
  
Superposition (or
 − −  Fredholm)
  integral of the
=  H  f ( ,  ) ( x −  , y −  )  d d  first kind
− −
 
=  f ( ,  ) H  ( x −  , y −  )  d d  Impulse
− −
response
6/22/2021 67
 Linear, Position-Invariant Degradations

Assume for a moment that  ( x, y) = 0

if H is a linear operator and position invariant,
H  ( x −  , y −  )  = h( x −  , y −  )
g ( x, y ) = H  f ( x, y ) 
 
=  f ( ,  ) H  ( x −  , y −  )  d d 
− − Convolution
  integral in 2-D
=  f ( ,  ) h( x −  , y −  ) d d 
− −

6/22/2021 68
 Linear, Position-Invariant Degradations

In the presence of additive noise,

if H is a linear operator and position invariant,

 
g ( x, y ) =   f ( ,  )h( x −  , y −  ) d d  +  ( x, y )
− −

= h ( x, y ) f ( x , y ) +  ( x, y )

G (u , v) = H (u, v) F (u, v) + N (u , v)

6/22/2021 69
 Estimating the Degradation Function

► Three principal ways to estimate the degradation function

1. Observation

2. Experimentation

3. Mathematical Modeling

6/22/2021 70
 Mathematical Modeling (1)

► Environmental conditions cause degradation

A model about atmospheric turbulence

− k ( u 2 + v 2 )5/6
H (u, v) = e
k : a constant that depends on
the nature of the turbulence

6/22/2021 71
6/22/2021 72
 Mathematical Modeling (2)

► Derive a mathematical model from basic principles

E.g., An image blurred by uniform linear motion between

the image and the sensor during image acquisition

6/22/2021 73
 Mathematical Modeling (3)

Suppose that an image f ( x, y) undergoes planar motion,

x0 (t ) and y0 (t ) are the time-varying components of motion
in the x- and y -directions, respectively.
The optical imaging process is perfect. T is the duration
of the exposure. The blurred image g ( x, y)
g ( x, y ) =  f  x − x0 (t ), y − y0 (t ) dt
T

6/22/2021 74
 Mathematical Modeling (4)

g ( x, y ) =  f  x − x0 (t ), y − y0 (t ) dt
T

0
 
G (u , v) =   g ( x, y ) e − j 2 ( ux + vy )
dxdy
− −


=    f  x − x0 (t ), y − y0 (t ) dt  e − j 2 (ux + vy ) dxdy
 T 

− − 
 0 

=
T
   f  x − x (t ), y − y (t )  e − j 2 (ux + vy ) dxdy dt
0  − − 0 0 
T − j 2 ux0 ( t ) + vy0 ( t ) 
=  F (u , v)e dt
0
T − j 2 ux0 ( t ) + vy0 ( t ) 
= F (u, v)  e dt
0
6/22/2021 75
 Mathematical Modeling (4)

T − j 2 ux0 ( t ) + vy0 ( t )
H (u, v) =  e dt
0

Suppose that the image undergoes uniform linear motion

in the x-direction only, at a rate given by x0 (t ) = at / T .
T
H (u, v) =  e − j 2 ux0 (t ) dt
0
T
=  e − j 2 uat /T dt
0

T
= sin( ua)e − j ua
 ua
6/22/2021 76
 Mathematical Modeling (5)

Suppose that the image undergoes uniform linear motion

in the x-direction and y -direction, at a rate given by
x0 (t ) = at / T and y0 (t ) = bt / T
T − j 2 ux0 ( t ) + vy0 ( t )
H (u, v) =  e dt
0
T
= e − j 2 [ ua + vb ]t / T
dt
0

T
= sin  (ua + vb)  e − j (ua + vb )
 (ua + vb)
6/22/2021 77
6/22/2021 78
 Inverse Filtering

An estimate of the transform of the original image

G (u , v)
F (u, v) =
H (u , v)

F (u , v) H (u , v) + N (u , v)
F (u , v) =
H (u , v)
N (u , v)
= F (u , v ) +
H (u , v)
6/22/2021 79
 Inverse Filtering

N (u, v)
F (u, v) = F (u, v) +
H (u, v)

1. We can't exactly recover the undegraded image

because N (u, v) is not known.
2. If the degradation function has zero or very
small values, then the ratio N (u, v) / H (u, v) could
easily dominate the estimate F (u, v).

6/22/2021 80
 Inverse Filtering

EXAMPLE

The image in Fig. 5.25(b) was inverse filtered using the

exact inverse of the degradation function that generated
that image. That is, the degradation function is
5/6
− k ( u − M /2 ) + ( v − N /2) 2 
2

H (u, v) = e  
, k = 0.0025

6/22/2021 81
 Inverse Filtering

One approach is to limit the filter frequencies to values near the origin.

EXAMPLE

The image in Fig. 5.25(b) was inverse filtered using the

exact inverse of the degradation function that generated
that image. That is, the degradation function is
5/6
− k ( u − M /2 ) + ( v − N /2) 2 
2

H (u, v) = e  

k = 0.0025, M = N = 480.
6/22/2021 82
 The poor performance
of direct inverse
filtering in general

A Butterworth
lowpass
function of
order 10

6/22/2021 83
 Drawback of Inverse Filtering

► It is not always possible to obtain an

inverse.

► It will not perform well in presence of noise.

If noise is present in the image, the inverse
filter will tend to amplify noise which is
undesirable.

6/22/2021 84
 Minimum Mean Square Error (Wiener)
Filtering
➢ N. Wiener (1942)

➢ Objective
Find an estimate of the uncorrupted image such that the mean
square error between them is minimized


e2 = E ( f − f )2 

6/22/2021 85
 Minimum Mean Square Error (Wiener)
Filtering

The minimum of the error function is given in the frequency domain

by the expression
 H *(u, v) S f (u, v) 
F (u , v ) =   G (u , v )
 S f (u , v) | H (u, v) | + S (u, v) 
2

 H *(u, v) 
=  G (u , v)
 | H (u, v) | + S (u, v ) / S f (u , v) 
2

 1 | H (u , v) |2 
=  G (u , v)
 H (u, v) | H (u, v) | + S (u, v) / S f (u, v) 
2

6/22/2021 86
 Minimum Mean Square Error (Wiener)
Filtering

 1 | H (u , v) |2 
F (u , v) =   G (u , v)
 H (u , v) | H (u , v) | + S (u , v) / S f (u , v) 
2

H (u , v) : degradation function
H *(u , v): complex conjugate of H (u, v)
| H (u , v) |2 = H *(u , v) H (u , v)
S (u , v) =| N (u , v) |2 = power spectrum of the noise
S f (u , v) =| F (u , v) |2 = power spectrum of the undegraded image

6/22/2021 87
 Minimum Mean Square Error (Wiener)
Filtering

 1 | H (u, v) |2 
F (u, v) =   G (u, v)
 H (u, v) | H (u, v) | + K 
2

K is a specified constant. Generally, the value of K

is chosen interactively to yield the best visual results.

6/22/2021 88
 Minimum Mean Square Error (Wiener)
Filtering

6/22/2021 89
Left:
degradated
image

Middle:
inverse
filtering

Right:
Wiener
filtering

6/22/2021 90
 Some Measures (1)

Singal-to-Noise Ratio (SNR)

M −1 N −1

 | F (u , v ) |2

SNR = u =0 v =0
M −1 N −1

 | N (u
u =0 v =0
, v ) |2

This ratio gives a measure of the level of information

bearing singal power to the level of noise power.

6/22/2021 91
 Some Measures (2)

Mean Square Error (MSE)

M −1 N −1 2
1
MSE=
MN
   f ( x, y) − f ( x, y) 
x =0 y =0

Root-Mean-Sqaure-Error (RMSE)
M −1 N −1

 f ( x, y ) 2
RMSE = M −1 N −1
u =0 v =0

 | f (
u =0 v =0
x , y ) − f ( x , y ) | 2

6/22/2021 92