Constrained matrix inversion 445

integer values in between. When i is incremented by 1 in the next row, all the values of i − k
are shifted by one position to the right (see equations (1.39) and (5.149) ). So, each partition
submatrix of H is characterised by the value of j − l ≡ u and has a circulant form:

⎛ ⎞
h(0, u) h(N − 1, u) h(N − 2, u) . . . h(1, u)
⎜ h(1, u) h(0, u) h(N − 1, u) . . . h(2, u)⎟
⎜ ⎟
⎜ h(2, u) h(1, u) h(0, u) . . . h(3, u)⎟
Hu ≡ ⎜ ⎟ (5.181)
⎜ .. .. .. .. ⎟
⎝ . . . . ⎠
h(N − 1, u) h(N − 2, u) h(N − 3, u) . . . h(0, u)

Notice that here we assume that h(v, u) is periodic with period N in each of its arguments,
and so h(1 − N, u) = h((1 − N ) + N, u) = h(1, u) etc.
The full matrix H may be written in the form

⎛ ⎞
H0 H−1 H−2 . . . H−M +1
⎜ H1 H0 H−1 . . . H−M +2 ⎟
⎜ ⎟
⎜ H2 H1 H0 . . . H−M +3 ⎟
H =⎜ ⎟ (5.182)
⎜ .. .. .. .. ⎟
⎝ . . . . ⎠
HM −1 HM −2 HM −3 ... H0

where again owing to the periodicity of h(v, u), H−1 = HM −1 , H−M +1 = H1 etc.

How can we diagonalise a block circulant matrix?

Define a matrix with elements

- .
1 2πj
WN (k, n) ≡ √ exp kn (5.183)
N N
and matrix

W ≡ WN ⊗ WN (5.184)

where ⊗ is the Kronecker product of the two matrices (see example 1.26, on page 38). The
inverse of WN (k, n) is a matrix with elements:

- .
1 2πj
WN−1 (k, n) = √ exp − kn (5.185)
N N

The inverse of W is given by (see example 5.28):

WN−1 (k, n) = WN−1 ⊗ WN−1 (5.186)

We also define a diagonal matrix Λ as

446 Image Processing: The Fundamentals

 ) k *
N 2 Ĥ modN (k), N if i = k
Λ(k, i) = (5.187)
0  k
if i =

where Ĥ is the discrete Fourier transform of the point spread function h:

N −1 N −1
1 vy
h(x, y)e−2πj ( N + N )
ux
Ĥ(u, v) = (5.188)
N2 x=0 y=0

It can be shown then, by direct matrix multiplication, that:

H = W ΛW −1 ⇒ H −1 = W Λ−1 W −1 (5.189)
Thus, H can be inverted easily since it has been written as the product of matrices the
inversion of which is trivial.

Box 5.4. Proof of equation (5.189)

First we have to find how an element H(f, g) of matrix H is related to the point spread
function h(x, y). Let us write indices f and g as multiples of the dimension N of one of
the partitions, plus a remainder:

f ≡ f1 N + f2
g ≡ g1 N + g2 (5.190)

As f and g scan all possible values from 0 to N 2 − 1 each, we can visualise the N × N
partitions of matrix H, indexed by subscript u ≡ f1 − g1 , as follows:

f1 = 0 f1 = 0 f1 = 0
g1 = 0 g1 = 1 g1 = 2
u = 0 u = −1 u = −2 . . .
f1 = 1 f1 = 1 f1 = 1
g1 = 0 g1 = 1 g1 = 2
u=1 u=0 u = −1 . . .
... ... ... ...

We observe that each partition is characterised by index u = f1 − g1 and inside each

partition the elements computed from h(x, y) are computed for various values of f2 − g2 .
We conclude that:
Constrained matrix inversion 447

H(f, g) = h(f2 − g2 , f1 − g1 ) (5.191)

−1
Let us consider next an element of matrix W ΛW :

N 2 −1 N 2 −1
−1
A(m, n) ≡ Wml Λlt Wtn (5.192)
l=0 t=0

Since matrix Λ is diagonal, the sum over t collapses to values of t = l only. Then:

N 2 −1
−1
A(m, n) = Wml Λll Wln (5.193)
l=0

Λll is a scalar and therefore it may change position inside the summand:

N 2 −1
−1
A(m, n) = Wml Wln Λll (5.194)
l=0
−1
In example 5.28 we saw how the elements of matrices Wml and Wln can be written if
we write their indices in terms of their quotients and remainders when divided by N :

m ≡ N m 1 + m2
l ≡ N l1 + l2
n ≡ N n1 + n2 (5.195)
Using these expressions and the definition of Λll as N 2 Ĥ(l2 , l1 ) from equation (5.187),
we obtain:

N 2 −1
2πj 2πj 1 − 2πj l1 n1 − 2πj l2 n2 2
A(m, n) = e N m 1 l1 e N m 2 l2 e N e N N Ĥ(l2 , l1 ) (5.196)
N2
l=0

On rearranging, we have:

N −1 N −1
2πj 2πj
A(m, n) = Ĥ(l2 , l1 )e N (m1 −n1 )l1 e N (m2 −n2 )l2 (5.197)
l1 =0 l2 =0

We recognise this expression as the inverse Fourier transform of

h(m2 − n2 , m1 − n1 ). Therefore:

A(m, n) = h(m2 − n2 , m1 − n1 ) (5.198)

By comparing equations (5.191) and (5.198) we can see that the elements of matrices H
and W ΛW −1 have been shown to be identical, and so equation (5.189) has been proven.
448 Image Processing: The Fundamentals

Box 5.5. What is the transpose of matrix H?

We shall show that H T = W Λ∗ W −1 , where Λ∗ is the complex conjugate of matrix Λ.

According to equation (5.191) of Box 5.4, an element of the transpose of matrix H will
be given by:

H T (f, g) = h(g2 − f2 , g1 − f1 ) (5.199)

(The roles of f and g are exchanged in the formula.) An element A(m, n) of matrix
W Λ∗ W −1 will be given by an equation similar to (5.197), but instead of having factor
Ĥ(l2 , l1 ), it will have factor Ĥ(−l2 , −l1 ), coming from the element of Λ∗ll being defined
in terms of the complex conjugate of the Fourier transform Ĥ(u, v) given by equation
(5.188):

N −1 N −1
2πj 2πj
A(m, n) = Ĥ(−l2 , −l1 )e N (m1 −n1 )l1 e N (m2 −n2 )l2 (5.200)
l1 =0 l2 =0

We change the dummy variables of summation to:

l̃1 ≡ −l1 and ˜l2 ≡ −l2 (5.201)

Then:

−N +1 −N +1
2πj 2πj
A(m, n) = Ĥ(˜l2 , l̃1 )e N (−m1 +n1 )l̃1 e N (−m2 +n2 )l̃2 (5.202)
l̃1 =0 l̃2 =0

Since we are dealing with periodic functions summed over a period, the range over which
we sum does not really matter, as long as N consecutive values are considered. Then
we can write:

N −1 N −1
2πj 2πj
A(m, n) = Ĥ(˜l2 , l̃1 )e N (−m1 +n1 )l̃1 e N (−m2 +n2 )l̃2 (5.203)
l̃1 =0 l̃2 =0

We recognise on the right-hand side of the above expression the inverse Fourier transform
of Ĥ(˜l2 , l̃1 ), computed at (n2 − m2 , n1 − m1 ):

A(m, n) = h(n2 − m2 , n1 − m1 ) (5.204)

By direct comparison with equation (5.199), we prove that matrices H T and W Λ∗ W −1

are equal, element by element.
Constrained matrix inversion 449

Example 5.29

Show that the Laplacian, ie the sum of the second derivatives, of a discrete
image at a pixel position (i, j) may be estimated by:

Δ2 f (i, j) = f (i − 1, j) + f (i, j − 1) + f (i + 1, j) + f (i, j + 1) − 4f (i, j) (5.205)

At inter-pixel position (i + 0.5, j), the first derivative of the image function along the
i axis is approximated by the first difference:

Δi f (i + 0.5, j) = f (i + 1, j) − f (i, j) (5.206)

Similarly, the first difference at (i − 0.5, j) along the i axis is:

Δi f (i − 0.5, j) = f (i, j) − f (i − 1, j) (5.207)

The second derivative at (i, j) along the i axis may be approximated by the first differ-
ence of the first differences, computed at positions (i + 0.5, j) and (i − 0.5, j), that is:

Δ2i f (i, j) = Δi f (i + 0.5, j) − Δi f (i − 0.5, j)

= f (i + 1, j) − 2f (i, j) + f (i − 1, j) (5.208)

Similarly, the second derivative at (i, j) along the j axis may be approximated by:

Δ2j f (i, j) = Δj f (i, j + 0.5) − Δj f (i, j − 0.5)

= f (i, j + 1) − 2f (i, j) + f (i, j − 1) (5.209)

Adding equations (5.208) and (5.209) by parts we obtain the result.

Example 5.30

Consider a 3 × 3 image represented by a column vector f . Identify a 9 × 9

matrix L such that if we multiply vector f by it, the output will be a vector
with the estimate of the value of the Laplacian at each position. Assume
that image f is periodic in each direction with period 3. What type of
matrix is L?

From example 5.29 we know that the point spread function of the operator that returns
the estimate of the Laplacian at each position is:
450 Image Processing: The Fundamentals

0 1 0
1 −4 1 (5.210)
0 1 0

To avoid boundary effects, we first extend the image in all directions periodically:

⎛f31 f32 f33 ⎞

f13 f11 f12 f13 f11
f21 ⎝f21 f22 f23 ⎠ f21 (5.211)
f33 f31 f32 f33 f31
f11 f12 f13

By observing which values will contribute to the value of the Laplacian at a pixel
position, and with what weight, we construct the 9 × 9 matrix with which we must
multiply the column vector f to obtain its Laplacian:

⎛ ⎞⎛ ⎞
−4 1 1 1 0 0 1 0 0 f11
⎜ 1 −4 1 0 1 0 0 1 0 ⎟ ⎜f21 ⎟
⎜ ⎟⎜ ⎟
⎜ 1 1 −4 0 0 1 0 0 1 ⎟ ⎜f31 ⎟
⎜ ⎟⎜ ⎟
⎜ 1 0 0 −4 1 1 1 0 0 ⎟ ⎜f12 ⎟
⎜ ⎟⎜ ⎟
⎜ 0 1 0 1 −4 1 0 1 0 ⎟ ⎜f22 ⎟ (5.212)
⎜ ⎟⎜ ⎟
⎜ 0 0 1 1 1 −4 0 0 1 ⎟ ⎜f32 ⎟
⎜ ⎟⎜ ⎟
⎜ 1 0 0 1 0 0 −4 1 1 ⎟ ⎜f13 ⎟
⎜ ⎟⎜ ⎟
⎝ 0 1 0 0 1 0 1 −4 1 ⎠ ⎝f23 ⎠
0 0 1 0 0 1 1 1 −4 f33

This matrix is a block circulant matrix with easily identifiable partitions of size 3 × 3.

Example B5.31

Using the matrix defined in example 5.30, estimate the Laplacian of the
following image:
⎛ ⎞
3 2 1
⎝2 0 1⎠ (5.213)
0 0 1

Then re-estimate the Laplacian of the above image using the formula of
example 5.29.
Constrained matrix inversion 451

⎛ ⎞⎛ ⎞ ⎛ ⎞
−4 1 1 1 0 0 1 0 0 3 −7
⎜ 1 −4 1 0 1 0 0 1 0 ⎟ ⎜2⎟ ⎜ −4 ⎟
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎜ 1 1 −4 0 0 1 0 0 1 ⎟ ⎜0⎟ ⎜ 6 ⎟
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎜ 1 0 0 −4 1 1 1 0 0 ⎟ ⎜2⎟ ⎜ −4 ⎟
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎜ 0 1 0 1 −4 1 0 1 0 ⎟ ⎜0⎟ = ⎜ 5 ⎟ (5.214)
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎜ 0 0 1 1 1 −4 0 0 1 ⎟ ⎜0⎟ ⎜ 3 ⎟
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎜ 1 0 0 1 0 0 −4 1 1 ⎟ ⎜1⎟ ⎜ 3 ⎟
⎜ ⎟⎜ ⎟ ⎜ ⎟
⎝ 0 1 0 0 1 0 1 −4 1 ⎠ ⎝1⎠ ⎝ 0 ⎠
0 0 1 0 0 1 1 1 −4 1 −2

If we use the formula, we need to augment first the image by writing explicitly the
boundary pixels:

0 0 1
⎛ ⎞
1 3 2 1 3
1 ⎝2 0 1⎠ 2 (5.215)
1 0 0 1 0
3 2 1

The Laplacian is:

⎛ ⎞ ⎛ ⎞
1+2+2−4×3 3+1−4×2 1+2+1+3−4×1 −7 −4 3
⎝ 3+1−4×2 2+2+1 1 + 1 + 2 − 4 × 1 ⎠ = ⎝ −4 5 0 ⎠
2+1+3 2+1 1+1−4×1 6 3 −2
(5.216)
Note that we obtain the same answer, whether we use the local formula or matrix
multiplication.

Example B5.32

Find the eigenvalues and eigenvectors of the matrix worked out in example
5.30.

Matrix L worked out in example 5.30 is:

452 Image Processing: The Fundamentals

⎛ ⎞
−4 1 1 1 0 0 1 0 0
⎜ 1 −4 1 0 1 0 0 1 0 ⎟
⎜ ⎟
⎜ 1 1 −4 0 0 1 0 0 1 ⎟
⎜ ⎟
⎜ 1 0 0 −4 1 1 1 0 0 ⎟
⎜ ⎟
L = ⎜
⎜ 0 1 0 1 −4 1 0 1 0 ⎟
⎟ (5.217)
⎜ 0 0 1 1 1 −4 0 0 1 ⎟
⎜ ⎟
⎜ 1 0 0 1 0 0 −4 1 1 ⎟
⎜ ⎟
⎝ 0 1 0 0 1 0 1 −4 1 ⎠
0 0 1 0 0 1 1 1 −4

This matrix is a block circulant matrix with easily identifiable 3 × 3 partitions. To

find its eigenvectors, we first use equation (5.152), on page 438, for M = 3 to define
vectors w:

⎛ ⎞ ⎛ ⎞ ⎛ ⎞
1 1 1
1 ⎝ ⎠ ⎝e ⎠2πj 1 ⎝ 4πj ⎠
w(0) = √ 1 w(1) = √1
3
3 w(2) = √ e 3 (5.218)
3 1 4πj 3 8πj
e 3 e 3

These vectors are used as columns to construct the matrix defined by equation (5.183):

⎛ ⎞
1 1 1
1 ⎝ 2πj 4πj
W3 = √ 1 e 3 e ⎠ 3 (5.219)
3 4πj 8πj
1 e 3 e 3

We take the Kronecker product of this matrix with itself to create matrix W as defined
by equation (5.184), on page 445:
⎛ ⎞
1 1 1 1 1 1 1 1 1
⎜1 e 2πj 4πj
e 3 1 e
2πj
e
4πj
1 e
2πj
e ⎟
4πj
⎜ 3 3 3 3 3
⎟
⎜1 e 4πj 8πj 4πj 8πj 4πj 8πj ⎟
⎜ 3 e 3 1 e 3 e 3 1 e 3 e 3 ⎟
⎜ 2πj 2πj 2πj 4πj 4πj 4πj ⎟
⎜1 1 1 e 3 e 3 e 3 e 3 e 3 e 3 ⎟
1⎜ 8πj ⎟
W = ⎜ e 3 ⎟
2πj 4πj 2πj 4πj 6πj 4πj 6πj

⎜ 1 e 3 e 3 e 3 e 3 e 3 e 3 e 3 ⎟ (5.220)
3⎜
e 3 ⎟
4πj 8πj 2πj 6πj 10πj 4πj 8πj 12πj
⎜1 e 3 e 3 e 3 e 3 e 3 e 3 e 3 ⎟
⎜ 4πj 4πj 4πj 8πj 8πj 8πj ⎟
⎜1 1 1 e 3 e 3 e 3 e 3 e 3 e 3 ⎟
⎜ 2πj 4πj 4πj 6πj 8πj 8πj 10πj 12πj ⎟
⎝1 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 3 ⎠
4πj 8πj 4πj 8πj 12πj 8πj 12πj 16πj
1 e 3 e 3 e 3 e 3 e 3 e 3 e 3 e 3

The columns of this matrix are the eigenvectors of matrix L. These eigenvectors are
the same for all block circulant matrices with the same structure, independent of what
the exact values of the elements are. The inverse of matrix W can be constructed using
equation (5.185), on page 445, ie by taking the complex conjugate of matrix W . (Note
that for a general unitary matrix we must take the complex conjugate of its transpose
Constrained matrix inversion 453

in order to construct its inverse. This is not necessary here as W is a symmetric

matrix and therefore it is equal to its transpose.)

⎛ ⎞
1 1 1 1 1 1 1 1 1
⎜1 2πj
e− 3
4πj
e− 3 1
2πj
e− 3
4πj
e− 3 1
2πj
e− 3 e− 3 ⎟
4πj
⎜ ⎟
⎜1 4πj
e− 3
8πj
e− 3
4πj
e− 3
8πj
e− 3
4πj
e− 3 e− 3 ⎟
8πj
⎜ 1 1 ⎟
⎜ 2πj 2πj 2πj 4πj 4πj 4πj ⎟
⎜1 1 1 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 ⎟
1 ⎜ 8πj ⎟
W = ⎜ − 2πj − 4πj
e− 3 ⎟
2πj 4πj 6πj 4πj 6πj
−1
1 e 3 e 3 e− 3 e− 3 e− 3 e− 3 e− 3 ⎟(5.221)
3⎜⎜1 e− 3 ⎟
4πj 8πj 2πj 6πj 10πj 4πj 8πj 12πj
⎜ e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 ⎟
⎜ 4πj 4πj 4πj 8πj 8πj 8πj ⎟
⎜1 1 1 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 ⎟
⎜ 2πj 4πj 4πj 6πj 8πj 8πj 10πj 12πj ⎟
⎝1 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 ⎠
4πj 8πj 4πj 8πj 12πj 8πj 12πj 16πj
1 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3 e− 3

The eigenvalues of matrix L may be computed from its Fourier transform, using equa-
tion (5.187), on page 446. First, however, we need to identify the kernel l(x, y) of the
operator represented by matrix L and take its Fourier transform L̂(u, v) using equation
(5.188), on page 446. From example 5.30 we know that the kernel function is:

0 1 0
1 −4 1 (5.222)
0 1 0
We can identify then the following values for the discrete function l(x, y):
l(0, 0) = −4, l(−1, −1) = 0, l(−1, 0) = 1, l(−1, 1) = 0
l(0, −1) = 1, l(0, 1) = 1, l(1, −1) = 0, l(1, 0) = 1, l(1, 1) = 0 (5.223)
However, these values cannot be directly used in equation (5.188), which assumes a
function h(x, y) defined with positive values of its arguments only. We therefore need
a shifted version of our kernel, one that puts the value −4 at the top left corner of the
matrix representation of the kernel. We can obtain such a version by reading the first
column of matrix L and wrapping it around to form a 3 × 3 matrix:
−4 1 1
1 0 0 (5.224)
1 0 0
Then we have:
l(0, 0) = −4, l(0, 1) = 1, l(0, 2) = 1, l(1, 0) = 1
l(2, 0) = 1, l(1, 1) = 0, l(1, 2) = 0, l(2, 1) = 0, l(2, 2) = 0 (5.225)
We can use these values in equation (5.188) to derive:
1$ 2πj 2πj 2πj 2πj
%
L̂(u, v) = −4 + e− 3 v + e− 3 2v + e− 3 u + e− 3 2u (5.226)
9
454 Image Processing: The Fundamentals

Formula (5.187) says that the eigenvalues of matrix L, which appear along the diagonal
of matrix Λ(k, i), are
) *the values of the Fourier transform L̂(u, v), computed for u =
mod3 (k) and v = k3 , where k = 0, 1, . . . , 8. These values may be computed using
formula (5.226):
L̂(0, 0) = 0
1$ 2πj 4πj
% 1 1
L̂(0, 1) = −4 + e− 3 + e− 3 + 1 + 1 = [−2 − 2 cos 60◦ ] = −
9 9 3
1$ − 4πj − 8πj
% 1
− 4πj
− 2πj 1
L̂(0, 2) = −4 + e 3 + e 3 + 2 = [−2 + e 3 + e 3 ] = −
9 9 3
L̂(1, 0) = L̂(0, 1) = −1
1$ 2πj 4πj
% 1 2
L̂(1, 1) = −4 + 2e− 3 + 2e− 3 = [−4 − 4 cos 60◦ ] = −
9 9 3
1$ − 4πj − 8πj
− 2πj
− 4πj
% 2
L̂(1, 2) = −4 + e 3 + e 3 + e 3 + e 3 = −
9 3
1
L̂(2, 0) = L̂(0, 2) = −
3
2
L̂(2, 1) = L̂(1, 2) = −
3
1$ 4πj 8πj
% 2
L̂(2, 2) = −4 + 2e− 3 + 2e− 3 = − (5.227)
9 3
Here we made use of the following:
√
− 2πj ◦ ◦ 1 3
e 3 = − cos 60 − j sin 60 = − − j
2 √2
4πj 1 3
e− 3 = − cos 60◦ + j sin 60◦ = − + j
2 2
6πj
e− 3 = 1 √
8πj
− 3 − 2πj ◦ ◦ 1 3
e = e 3 = − cos 60 − j sin 60 = − − j (5.228)
2 2
Note that the first eigenvalue of matrix L is 0. This means that matrix L is singular,
and even though we can diagonalise it using equation (5.189), we cannot invert it
by taking the inverse of this equation. This should not be surprising as matrix L
expresses the Laplacian operator on an image, and we know that from the knowledge
of the Laplacian alone we can never recover the original image.
Applying equation (5.187) we define matrix ΛL for L to be:
⎡ ⎤
0 0 0 0 0 0 0 0 0
⎢ 0 −3 0 0 0 0 0 0 0 ⎥
⎢ ⎥
⎢ 0 0 −3 0 0 0 0 0 0 ⎥
⎢ ⎥
⎢ 0 0 0 −3 0 0 0 0 0 ⎥
⎢ ⎥
ΛL = ⎢ ⎢ 0 0 0 0 −6 0 0 0 0 ⎥⎥ (5.229)
⎢ 0 0 0 0 0 −6 0 0 0 ⎥
⎢ ⎥
⎢ 0 0 0 0 0 0 −3 0 0 ⎥
⎢ ⎥
⎣ 0 0 0 0 0 0 0 −6 0 ⎦
0 0 0 0 0 0 0 0 −6
Constrained matrix inversion 455

Having defined matrices W , W −1 and Λ we can then write:

L = W ΛL W −1 (5.230)

This equation may be confirmed by direct substitution. First we compute matrix

ΛL W −1 :

⎡ ⎤
0 0 0
0 0 0 0 0 0
⎢ −1 −e− 2πj
−e − 4πj
−1 −e − 2πj
−e − 4πj
−1 −e− 2πj
−e − 4πj ⎥
⎢ 3 3 3 3 3 3
⎥
⎢ −1 4πj
−e− 3
8πj
−e− 3 −1 −e− 3
4πj 8πj
−e− 3 −1
4πj
−e− 3
8πj
−e− 3 ⎥
⎢ ⎥
⎢ 2πj 2πj 2πj 4πj 4πj 4πj ⎥
⎢ −1 −1 −1 −e− 3 −e− 3 −e− 3 −e− 3 −e− 3 −e− 3 ⎥
⎢ ⎥
⎢ −2 2πj 4πj 2πj 4πj
−2e− 3 −2e− 3 −2e− 3 −2e− 3
6πj
−2e− 3
4πj
−2e− 3
6πj
−2e− 3 −2e− 3
8πj
⎥
⎢ ⎥
⎢ −2 4πj 8πj 2πj 6πj
−2e− 3 −2e− 3 −2e− 3 −2e− 3
10πj
−2e− 3
4πj
−2e− 3
8πj 12πj
−2e− 3 −2e− 3 ⎥
⎢ ⎥
⎢ 4πj 4πj 4πj 8πj 8πj 8πj ⎥
⎢ −1 −1 −1 −e− 3 −e− 3 −e− 3 −e− 3 −e− 3 −e− 3 ⎥
⎢ 2πj 4πj 4πj 6πj 8πj 8πj 10πj 12πj ⎥
⎣ −2 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 ⎦
4πj 8πj 4πj 8πj 12πj 8πj 12πj 16πj
−2 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3 −2e− 3

If we take into consideration that

√
− 10πj − 4πj ◦ ◦ 1 3
e 3 = e 3 = − cos 60 + j sin 60 = − + j
2 2
− 12πj
e 3 = 1 √
− 16πj − 4πj ◦ ◦ 1 3
e 3 = e 3 = − cos 60 + j sin 60 = − + j (5.231)
2 2
and multiply the above matrix with W from the left, we recover matrix L.

How can we overcome the extreme sensitivity of matrix inversion to noise?

We can do it by imposing a smoothness constraint to the solution, so that it does not fluctuate
too much. Let us say that we would like the second derivative of the reconstructed image to
be small overall. At each pixel, the sum of the second derivatives of the image along each
axis, known as the Laplacian, may be approximated by Δ2 f (i, k) given by equation (5.205)
derived in example 5.29. The constraint we choose to impose then is for the sum of the
squares of the Laplacian values at each pixel position to be minimal:

N −1 N −1
1 2 22
Δ f (i, k) = minimal (5.232)
k=0 i=0

The value of the Laplacian at each pixel position may be computed by using the Laplacian
operator which has the form of an N 2 × N 2 matrix acting on column vector f (of size N 2 × 1),
456 Image Processing: The Fundamentals

T
Lf . Lf is a vector. The sum of the squares of its elements are given by (Lf ) Lf . The
constraint then is:
(Lf )T Lf = minimal (5.233)

How can we incorporate the constraint in the inversion of the matrix?

Let us write again in matrix form the equation we want to solve for f :
g = Hf + ν (5.234)
We assume that the noise vector ν is not known but some of its statistical properties are
known; say we know that:
νT ν = ε (5.235)
This quantity ε is related to the variance of the noise and it could be estimated from the image
itself using areas of uniform brightness only. If we substitute ν from (5.234) into (5.235), we
have:
(g − Hf )T (g − Hf ) = ε (5.236)
The problem then is to minimise (5.233) under the constraint (5.236). The solution of
this problem is a filter with Fourier transform (see Box 5.6, on page 459, and example 5.36):
Ĥ ∗ (u, v)
M̂ (u, v) = 2 2 (5.237)
|Ĥ(u, v)| + γ|L̂(u, v)|
By multiplying numerator and denominator with Ĥ(u, v), we can bring this filter into a
form directly comparable with the inverse and the Wiener filters:
2
1 |Ĥ(u, v)|
M̂ (u, v) = × 2 2 (5.238)
Ĥ(u, v) |Ĥ(u, v)| + γ|L̂(u, v)|
Here γ is a constant and L̂(u, v) is the Fourier transform of an N × N matrix L, with the
following property: if we use it to multiply the image (written as a vector) from the left,
the output will be an array, the same size as the image, with an estimate of the value of the
Laplacian at each pixel position. The role of parameter γ is to strike the balance between
smoothing the output and paying attention to the data.

Example B5.33
If f is an N × 1 real vector and A is an N × N matrix, show that

∂f T Af
= (A + AT )f (5.239)
∂f
Constrained matrix inversion 457

Using the results of example 3.65, on page 269, we can easily see that:

∂f T Af ∂f T (Af ) ∂(f T A)f

= +
∂f ∂f ∂f
T
∂(AT f ) f
= Af +
∂f
= Af + AT f = (A + AT )f (5.240)

Here we made use of the fact that Af and AT f are vectors.

Example B5.34

If g is the column vector that corresponds to a 3 × 3 image G and matrix

W −1 is defined as in example 5.28 for N = 3, show that vector W −1 g is
proportional to the discrete Fourier transform Ĝ of G.

Assume that:
⎛ ⎞ ⎛ ⎞
g11 g12 g13 1 1 1
1 2πj 2πj
G = ⎝g21 g22 g23 ⎠ and W3−1 = √ ⎝1 e− 3 e− 3 2 ⎠ (5.241)
3 2πj 2πj
g31 g32 g33 1 e− 3 2 e− 3

Then:

W −1 = W3−1 ⊗ W3−1 =
⎛ ⎞
1 1 1 1 1 1 1 1 1
⎜1 e− 2πj e− 2πj
3 2 1
2πj
e− 3
2πj
e− 3 2 1
2πj
e− 3 e− 3 2 ⎟
2πj
⎜ 3
⎟
⎜1 e− 2πj
3 2 e− 3
2πj 2πj
e− 3 2
2πj
e− 3 e− 3 2 e− 3 ⎟
2πj 2πj
⎜ 1 1 ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj ⎟
⎜1 1 1 e− 3 e− 3 e− 3 e− 3 2 e− 3 2 e− 3 2 ⎟
⎜
1⎜ 2πj ⎟
e− 3 2 e− 3 3 e− 3 4 ⎟
2πj 2πj 2πj 2πj 2πj 2πj 2πj
1 e− 3 e− 3 2 e− 3 e− 3 2 e− 3 3 ⎟ (5.242)
3⎜
⎜1 e− 3 2 e− 3 e− 3 2 e− 3 4 e− 3 3 ⎟
2πj 2πj 2πj 2πj 2πj 2πj 2πj 2πj
⎜ e− 3 e− 3 3 e− 3 2 ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj ⎟
⎜1 1 1 e− 3 2 e− 3 2 e− 3 2 e− 3 e− 3 e− 3 ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj 2πj 2πj ⎟
⎝1 e− 3 e− 3 2 e− 3 2 e− 3 3 e− 3 4 e− 3 e− 3 2 e− 3 3 ⎠
2πj 2πj 2πj 2πj 2πj 2πj 2πj 2πj
1 e− 3 2 e− 3 e− 3 2 e− 3 4 e− 3 3 e− 3 e− 3 3 e− 3 2

2πj 2πj 2πj

3 − 2πj 2πj
If we use e− 3 3
= e−2πj = 1 and e− 3 4 = e− 3 e 3 = e− 3 , this matrix simpli-
458 Image Processing: The Fundamentals

fies somehow. So we get:

W −1 g =

⎛ ⎞
1 1 1 1 1 1 1 1 1 ⎛ ⎞
⎜1 e− 2πj − 2πj 2 − 2πj − 2πj 2 − 2πj − 2πj 2 ⎟ g11
⎜ 3 e 3 1 e 3 e 3 1 e 3 e 3
⎟⎜g21 ⎟
⎜1 e− 2πj
3 2 e− 3
2πj 2πj
e− 3 2 e− 3
2πj
e− 3 2 e− 3 ⎟
2πj 2πj ⎜ ⎟
⎜ 1 1 ⎟⎜g ⎟
⎜ − 2πj
− 2πj
− 2πj
− 2πj
− 2πj
− 2πj ⎟⎜ 31 ⎟
⎜1 1 1 e 3 e 3 e 3 e 3 2
e 3 2
e 3 ⎟⎜g12 ⎟
2
1⎜
⎜1 e− 2πj 2πj ⎟⎜ ⎟
e− 3 ⎟ ⎜g22 ⎟ =
2πj 2πj 2πj 2πj
− 2 − − 2 − 2
⎜ 3 e 3 e 3 e 3 1 e 3 1 ⎟ ⎜ ⎟
3⎜ − 2πj 2
1 ⎟ ⎜ ⎟
2πj 2πj 2πj 2πj 2πj
⎜1 e 3 e− 3 e− 3 1 e− 3 2 e− 3 2 e− 3 ⎟⎜g32 ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj ⎟⎜ ⎟
⎜1 1 1 e− 3 2 e− 3 2 e− 3 2 e− 3 e− 3 e− 3 ⎟⎜g13 ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj ⎟ ⎝g ⎠
⎝1 e− 3 e− 3 2 e− 3 2 1 e− 3 e− 3 e− 3 2 1 ⎠ 23
2πj 2πj 2πj 2πj 2πj 2πj g33
1 e− 3 2 e− 3 e− 3 2 e− 3 1 e− 3 1 e− 3 2

⎛ ⎞
g11+g21+g31+g12+g22+g32+g13+g23+g33
⎜ ⎟
⎜ 2πj 2πj 2πj 2πj 2πj 2πj ⎟
⎜g11+g21 e− 3 +g31 e− 3 2+g12+g22 e− 3 +g32 e− 3 2+g13+g23 e− 3 +g33 e− 3 2⎟
1⎜
⎜ ..
⎟
⎟
⎜
3⎜ . ⎟
⎟
⎜g +g +g +g e− 2πj 3 2+g
2πj
− 3 2
+g32 e− 3 2+g13 e− 3 +g23 e− 3 +g33 e− 3 ⎟
2πj 2πj 2πj 2πj
⎝ 11 21 31 12 22 e ⎠
..
.
(5.243)

Careful examination of the elements of this vector shows that they are the Fourier
components of G, multiplied with 3, computed at various combinations of frequencies
(u, v), for u = 0, 1, 2 and v = 0, 1, 2, and arranged as follows:

⎛ ⎞
Ĝ(0, 0)
⎜Ĝ(1, 0)⎟
⎜ ⎟
⎜Ĝ(2, 0)⎟
⎜ ⎟
⎜Ĝ(0, 1)⎟
⎜ ⎟
⎜ ⎟
3 × ⎜Ĝ(1, 1)⎟ (5.244)
⎜ ⎟
⎜Ĝ(2, 1)⎟
⎜ ⎟
⎜Ĝ(0, 2)⎟
⎜ ⎟
⎝Ĝ(1, 2)⎠
Ĝ(2, 2)

This shows that W −1 g yields N times the Fourier transform of G, as a column vector.
Constrained matrix inversion 459

Example B5.35

Show that, if matrix Λ is defined by equation (5.187), then Λ∗ Λ is a diagonal

2
) k * along the diagonal being N |Ĥ(k2 , k1 )| , where
matrix with its kth element 4

k2 ≡ modN (k) and k1 ≡ N .

From the definition of Λ, equation (5.187), we can write:

⎛ 2 ⎞
N Ĥ(0, 0) 0 0 ... 0
⎜ 0 N 2
Ĥ(1, 0) 0 ... 0 ⎟
⎜ ⎟
⎜ 0 0 2
N Ĥ(2, 0) . . . 0 ⎟
Λ=⎜ ⎟ (5.245)
⎜ .. .. .. .. ⎟
⎝ . . . . ⎠
0 0 0 . . . N 2 Ĥ(N − 1, N − 1)

Then:
⎛ 2 ∗ ⎞
N Ĥ (0, 0) 0 0 ... 0
⎜ 0 N 2 ∗
Ĥ (1, 0) 0 ... 0 ⎟
⎜ ⎟
∗ ⎜ 0 0 2 ∗
N Ĥ (2, 0) . . . 0 ⎟
Λ =⎜ ⎟(5.246)
⎜ .. .. .. .. ⎟
⎝ . . . . ⎠
0 0 0 . . . N 2 Ĥ ∗ (N − 1, N − 1)

Obviously:
⎛ 2
⎞
N 4 |Ĥ(0, 0)| 0 0 ... 0
⎜ 2 ⎟
⎜ 0 N 4 |Ĥ(1, 0)| 0 ... 0 ⎟
⎜ ⎟
∗ ⎜ 2 ⎟
Λ Λ=⎜ 0 0 N 4 |Ĥ(2, 0)| ... 0 ⎟
⎜ ⎟
⎜ .. .. .. .. ⎟
⎝ . . . . ⎠
2
0 0 0 . . . N 4 |Ĥ(N − 1, N − 1)|
(5.247)

Box 5.6. Derivation of the constrained matrix inversion filter

We must find the solution of the problem: minimise (Lf )T Lf with the constraint:

T
[g − Hf ] [g − Hf ] = ε (5.248)
460 Image Processing: The Fundamentals

According to the method of Lagrange multipliers (see Box 3.8, on page 268), the
solution must satisfy

∂ $ T T %
f L Lf + λ(g − Hf )T (g − Hf ) = 0 (5.249)
∂f
where λ is a constant. This differentiation is with respect to a vector and it will yield
a system of N 2 equations (one for each component of vector f ) which, with equation
(5.248), form a system of N 2 +1 equations, for the N 2 +1 unknowns: the N 2 components
of f plus λ.
If a is a vector and b another one, then it can be shown (example 3.65, page 267) that:

∂f T a
=a (5.250)
∂f

∂bT f
=b (5.251)
∂f
Also, if A is an N 2 × N 2 square matrix, then (see example 5.33, on page 456):

∂f T Af
= (A + AT )f (5.252)
∂f
We apply equations (5.250), (5.251) and (5.252) to (5.249) to perform the differentiation:

∂ $ %
f T (LT L)f +λ(gT g− gT Hf − f T HT g + fT HT Hf ) = 0
∂f         
eqn(5.252) eqn(5.251) eqn(5.250) eqn(5.252) with
with b ≡ H g with a ≡ H g A ≡ H H
T
with A ≡ L LT T T

⇒ (2LT L)f + λ(−H T g − H T g + 2H T Hf ) = 0

⇒ (H T H + γLT L)f = H T g (5.253)

Here γ ≡ λ1 . Equation (5.253) can easily be solved in terms of block circulant matrices.
Then:

−1
f = [H T H + γLT L] HT g (5.254)

Parameter γ may be specified by substitution in equation (5.248).

Constrained matrix inversion 461

Example B5.36

Solve equation (5.253).

Since H and L are block circulant matrices (see examples 5.30 and 5.32, and Box 5.5,
on page 448), they may be written as:
H = W ΛH W −1 H T = W Λ∗H W −1
L = W ΛL W −1 LT = W Λ∗L W −1 (5.255)
Then:
H T H + γLT L = W Λ∗H W −1 W ΛH W −1 + γW Λ∗L W −1 W ΛL W −1
= W Λ∗H ΛH W −1 + γW Λ∗L ΛL W −1
= W (Λ∗H ΛH + γΛ∗L ΛL )W −1 (5.256)
We substitute from (5.255) and (5.256) into (5.253) to obtain:
W (Λ∗H ΛH + γΛ∗L ΛL )W −1 f = W Λ∗H W −1 g (5.257)
−1
First we multiply both sides of the equation from the left with W , to get:
(Λ∗H ΛH + γΛ∗L ΛL )W −1 f = Λ∗H W −1 g (5.258)
Notice that as Λ∗H , Λ∗H ΛH and Λ∗L ΛL are diagonal matrices, this equation expresses
a relationship between the corresponding elements of vectors W −1 f and W −1 g one by
one.
Applying the result of example 5.35, we may write
2 2
Λ∗H ΛH = N 4 |Ĥ(u, v)| and Λ∗L ΛL = N 4 |L̂(u, v)| (5.259)
where L̂(u, v) is the Fourier transform of matrix L. Also, by applying the results of
example 5.34, we may write:
W −1 f = N F̂ (u, v) and W −1 g = N Ĝ(u, v) (5.260)
Finally, we replace Λ∗H by its definition, equation (5.187), so that (5.258) becomes:
$ 2 2
%
N 4 |Ĥ(u, v)| + γ|L̂(u, v)| N F̂ (u, v) = N 2 Ĥ ∗ (u, v)N Ĝ(u, v) ⇒
2 2
|Ĥ(u, v)| + γ|L̂(u, v)|
N2 F̂ (u, v) = Ĝ(u, v) (5.261)
Ĥ ∗ (u, v)
Note that when we work fully in the discrete domain, we have to use the form of the
convolution theorem that applies to DFTs (see equation (2.208), on page 108). Then
the correct form of equation (5.3), on page 396, is Ĝ(u, v) = N 2 Ĥ(u, v)Ĝ(u, v). This
means that the filter with which we have to multiply the DFT of the degraded image
in order to obtain the DFT of the original image is given by equation (5.237).
462 Image Processing: The Fundamentals

What is the relationship between the Wiener filter and the constrained matrix
inversion filter?

Both filters look similar (see equations (5.125) and (5.238)), but they differ in many ways.
1. The Wiener filter is designed to optimise the restoration in an average statistical sense
over a large ensemble of similar images. The constrained matrix inversion deals with
one image only and imposes constraints on the solution sought.
2. The Wiener filter is based on the assumption that the random fields involved are homo-
geneous with known spectral densities. In the constrained matrix inversion it is assumed
that we know only some statistical property of the noise.
In the constrained matrix restoration approach, various filters may be constructed using
the same formulation, by simply changing the smoothing criterion. For example, one may try
to minimise the sum of the squares of the first derivatives at all positions as opposed to the
second derivatives. The only difference from formula (5.237) will be in matrix L.

Example B5.37

Calculate the DFT of the N × N matrix L defined as:

⎛ ⎞
−4 1 1 ... 1
⎜ 1 0 0 ... 0 ⎟
⎜ ⎟
⎜ 0 0 0 ... 0 ⎟
⎜ ⎟
L ≡⎜ . .. .. .. ⎟ (5.262)
⎜ .. . . ... . ⎟
⎜ ⎟
⎝ 0 0 0 ... 0 ⎠
1 0 0 ... 0

By applying formula (5.188) for L (x, y), we obtain:

N −1 N −1
1 vy
L (x, y)e−2πj ( N + N )
ux
L̂ (u, v) =
N2 x=0 y=0
3 N −1
4
1 −2πj ux −2πj v
−2πj (N −1)v
= −4 + e N + e N + e N
N2 x=1
3 N −1
4
1 −2πj ux −2πj v
−2πj (N −1)v
= −4 + e N − 1 + e N + e N
N2 x=0
1 $ (N −1)v
%
−5 + N δ(u) + e−2πj N + e−2πj N
v
= 2
(5.263)
N
Here we made use of the geometric progression formula (2.165), on page 95.
Constrained matrix inversion 463

Example B5.38

Calculate the magnitude of the DFT of matrix L defined by equation

(5.262).

The real and the imaginary parts of the DFT computed in example 5.37 are:

- .
1 2πn 2π(N − 1)n
L1 (m, n) ≡ −5 + N δ(m) + cos + cos
N2 N N
- .
1 2πn 2π(N − 1)n
L2 (m, n) ≡ − sin − sin (5.264)
N2 N N

Then:

⎧ $ %
⎪
⎪ 1
N − 5 + cos 2πn 2π(N −1)n
m = 0, n = 0, 1, . . . , N − 1
⎨ N2 N + cos N
L1 (m, n) = $ %
⎪
⎪
⎩ 1
−5 + cos 2πn 2π(N −1)n
m = 0, n = 0, 1, . . . , N − 1
N2 N + cos N
(5.265)
Then:

-
1 2πn
L21 (0, n) + L22 (0, n) = (N − 5)2 + 2 + 2(N − 5) cos
N4 N
2π(N − 1)n 2πn 2π(N − 1)n
+2(N − 5) cos + 2 cos cos
N . N N
2πn 2π(N − 1)n
+2 sin sin
N N
-
1 2πn 2π(N − 1)n
= 4
(N − 5)2 + 2(N − 5) cos + 2(N − 5) cos
N N N
.
2π(N − 2)n
+2 cos
N
(5.266)

And:

- .
1 2πn 2π(N − 1)n 2π(N − 2)n
L21 (m, n)+L22 (m, n)= 25 + 2 − 10 cos − 10 cos + 2 cos
N4 N N N
(5.267)
464 Image Processing: The Fundamentals

How do we apply constrained matrix inversion in practice?

Apply the following algorithm.

Step 0: Select a smoothing operator and compute |L̂(u, v)|2 . If you select to use the Lapla-
cian, use formulae (5.266) and (5.267) to compute |L̂(u, v)|2 .
Step 1: Select a value for parameter γ. It has to be higher for higher levels of noise in
the image. The rule of thumb is that γ should be selected such that the two terms in the
denominator of (5.237) are roughly of the same order of magnitude.
Step 2: Compute the mean grey value of the degraded image.
Step 3: Compute the DFT of the degraded image.
Step 4: Multiply the DFT of the degraded image with function M̂ (u, v) of equation (5.237),
point by point.
Step 5: Take the inverse DFT of the result.
Step 6: Add the mean grey value of the degraded image to all elements of the result, to
obtain the restored image.

Example 5.39

Restore the images of figures 5.5a, on page 414, and 5.9a and 5.9b, on page
418, using constrained matrix inversion.

We must first define matrix L̂(u, v) which expresses the constraint.

Following the steps of example 5.29, on page 449, we can see that matrix L(i, j), with
which we have to multiply an N × N image in order to obtain the value of the
Laplacian at each position, is given by an N 2 × N 2 matrix of the following structure:
N-1 unit matrices NxN

NxN
matrix L
...
NxN
matrix L
...
N-1
...

...

...

unit
matrices
NxN

... NxN
matrix L

Matrix L̃ has the following form:

Constrained matrix inversion 465

N −3 zeros
⎛    ⎞
−4 1 0 0 ... 0 1
⎜ 1 −4 1 0 ... 0 0 ⎟
⎜ ⎟
⎜ 0 1 −4 1 ... 0 0 ⎟
⎜ ⎟
⎜ 0 0 1 −4 . . . 0 0 ⎟
⎜ ⎟
L̃ = ⎜ 0 0 0 1 ... 0 0 ⎟ (5.268)
⎜ ⎟
⎜ .. .. .. .. .. ..⎟
⎜ . . . . ... . .⎟
⎜ ⎟
⎝ 0 0 0 0 . . . −4 1 ⎠
1 0 0 0 ... 1 −4
To form the kernel we require, we must take the first column of matrix L and wrap it
to form an N × N matrix. The first column of matrix L consists of the first column
of matrix L̃ (N elements) plus the first columns of N − 1 unit matrices of size N × N .
These N 2 elements have to be written as N columns of size N next to each other, to
form an N × N matrix L , say:

⎛ ⎞
−4 1 1 . . . 1
⎜ 1 0 0 ... 0 ⎟
⎜ ⎟
⎜ 0 0 0 ... 0 ⎟
⎜ ⎟
L =⎜ .. .. .. ..⎟ (5.269)
⎜ . . . ... .⎟
⎜ ⎟
⎝ 0 0 0 ... 0 ⎠
1 0 0 ... 0
It is the Fourier transform of this matrix that appears in the constrained matrix inver-
sion filter. This Fourier transform may be computed analytically easily (see examples
5.37 and 5.38). Note that

2
|L̂(m, n)| = L21 (m, n) + L22 (m, n) (5.270)
where L̂ is the Fourier transform of L . This quantity has a factor 1/1284 . Let us omit
it, as it may be easily incorporated in the constant γ that multiplies it. For simplicity,
let us call this modified function A(m, n). From example 5.38, A(m, n) is given by:
⎧
⎪
⎪ 15131 + 246 cos 2πn 2π127n 2π126n
128 + 246 cos 128 + 2 cos 128 m=0
⎪
⎪
⎪
⎪ n = 0, 1, . . . , 127
⎨
A(m, n) ≡
⎪
⎪
⎪
⎪
⎪
⎪ 128 − 10 cos 128 + 2 cos 128
27 − 10 cos 2πn 2π127n 2π126n
m = 1, 2, . . . , 127
⎩
n = 0, 1, . . . , 127
(5.271)
The frequency response function of the filter we must use is then given by substituting
the frequency response function (5.50), on page 410, into equation (5.237):
466 Image Processing: The Fundamentals

1
iT sin πm sin iTNπm
N (iT −1)
πm
M̂ (m, n) = i πm
N
ej (5.272)
sin2 TN
i2T sin2 πm
+ γA(m, n)
N

For m = 0 we must use:

1
M̂ (0, n) = for 0≤n≤N −1 (5.273)
1 + γA(0, n)

Note from equation (5.271) that A(0, 0) is much larger than 1, making the dc com-
ponent of filter M̂ virtually 0. So, when we multiply the DFT of the degraded image
with M̂ , we kill its dc component. This is because the constraint we have imposed did
not have a dc component. To restore, therefore, the dc component of the image, after
filtering, we have to compute the dc component of the input image and add it to the
result, before we visualise it as an image.
Working as for the case of the Wiener filtering, we can work out that the real and
imaginary parts of the Fourier transform of the original image are given by:

$ %
iT sin πm iT πm
N sin N G1 (m, n) cos (iT −1)πm
N − G2 (m, n) sin (iT −1)πm
N
F1 (m, n) =
sin2 iTNπm + γA(m, n)i2T sin2 πm
N

$ %
iT sin πm iT πm
N sin N G1 (m, n) sin (iT −1)πm
N + G2 (m, n) cos (iT −1)πm
N
F2 (m, n) =
sin2 iTNπm + γA(m, n)i2T sin2 πm
N
(5.274)

These formulae are valid for 0 < m ≤ N − 1 and 0 ≤ n ≤ N − 1. For m = 0 we must

use formulae:

G1 (0, n)
F1 (0, n) =
1 + γA(0, n)

G2 (0, n)
F2 (0, n) = (5.275)
1 + γA(0, n)

If we take the inverse Fourier transform using functions F1 (m, n) and F2 (m, n) as the
real and the imaginary parts, and add the dc component, we obtain the restored image.
The results of restoring images 5.5b, 5.9a and 5.9b are shown in figure 5.14. Note
that different values of γ, ie different levels of smoothing, have to be used for different
levels of noise in the image.
Constrained matrix inversion 467

Input: image 5.5b Input: image 5.9a Input: image 5.9b

γ = 0.001, M SE = 1749 γ = 0.001, M SE = 3186 γ = 0.001, M SE = 6489

γ = 0.002, M SE = 1617 γ = 0.004, M SE = 1858 γ = 0.006, M SE = 2312

γ = 0.005, M SE = 1543 γ = 0.007, M SE = 1678 γ = 0.010, M SE = 1934

γ = 0.01, M SE = 1530 γ = 0.02, M SE = 1593 γ = 0.0999, M SE = 2144

Figure 5.14: Image restoration with constrained matrix inversion.
468 Image Processing: The Fundamentals

5.4 Inhomogeneous linear image restora-

tion: the whirl transform
How do we model the degradation of an image if it is linear but inhomogeneous?

In the general case, equation (1.15), on page 13, applies:

N N
g(i, j) = f (k, l)h(k, l, i, j) (5.276)
k=1 l=1

We have shown in Chapter 1 that this equation can be written in matrix form (see equation
(1.38), on page 19):

g = Hf (5.277)
For inhomogeneous linear distortions, matrix H is not circulant or block circulant. In order
to solve system (5.277) we can no longer use filtering. Instead, we must solve it by directly
inverting matrix H. However, this will lead to a noisy solution, so some regularisation process
must be included.

Example 5.40

In a notorious case in 2007, a criminal was putting on the Internet images

of himself while committing crimes, with his face scrambled in a whirl
pattern. Work out the distortion he might have been applying to the
subimage of his face.

First we have to create a whirl scanning pattern. This may be given by coordinates
(x, y) defined as

x(t) = x0 + αt cos(βt)
y(t) = y0 + αt sin(βt) (5.278)

where (x0 , y0 ) is the “eye” of the whirl, ie its starting point, t is a parameter incre-
mented along the scanning path, and α and β are parameters that define the exact
shape of the whirl. For example, for a tight whirl pattern, α must be small. The in-
teger coordinates (i, j) of the image that will make up the scanning path will be given
by

i = i0 + αt cos(βt) + 0.5

j = j0 + αt sin(βt) + 0.5 (5.279)
Inhomogeneous linear image restoration 469

where (i0 , j0 ) are the coordinates of the starting pixel, and α and β are chosen to be
much smaller than 1. Parameter t is allowed to take positive integer values starting
from 0. Once we have the sequence of pixels that make up the scanning pattern, we may
smear their values by, for example, averaging the previous K values and assigning the
result to the current pixel of the scanning sequence. For example, if the values of three
successive pixels are averaged and assigned to the most recent pixel in the sequence,
(K = 3), the values of the scrambled image g̃ will be computed according to:

g̃ (i0 + αt cos(βt) + 0.5 , j0 + αt sin(βt) + 0.5) =

1
{g (i0 + α(t − 2) cos[β(t − 2)] + 0.5 , j0 + α(t − 2) sin[β(t − 2)] + 0.5) +
3
g (i0 + α(t − 1) cos[β(t − 1)] + 0.5 , j0 + α(t − 1) sin[β(t − 1)] + 0.5) +
g (i0 + αt cos[βt] + 0.5 , j0 + αt sin[βt] + 0.5)} (5.280)

Example 5.41

Use the scrambling pattern of example 5.40 to work out the elements of
matrix H with which one should operate on an M × N image in order to
scramble it in a whirl-like way. Assume that in the scrambling pattern the
values of K + 1 successive pixels are averaged.

Remember that the H mapping should be of size M N × M N , because it will operate

on the image written as a column vector with its columns written one under the other.
To compute the elements of this matrix we apply the following algorithm.

Step 1: Create an array H of size M N × M N with all its elements 0.

Step 2: Choose (i0 , j0 ) to be the coordinates of a pixel near the centre of the image.
2π
Step 3: Select values for α and β, say α = 0.1 and β = 360 5. Select the maximum
value of t you will use, say tmax = 10, 000.
Step 4: Create a 1D array S of M N samples, all with flag 0. This array will be used
to keep track which rows of matrix H have all their elements 0.
Step 5: Starting with t = 0 and carrying on with t = 1, 2, . . . , tmax , or until all
elements of array S have their flag down, perform the following computations.
Compute the indices of the 2D image pixels that will have to be mixed to yield the
value of output pixel (ic , jc ):

ic = i0 + αt cos(βt) + 0.5 jc = j0 + αt sin(βt) + 0.5

i1 = i0 + α(t − 1) cos[β(t − 1)] + 0.5 j1 = j0 + α(t − 1) sin[β(t − 1)] + 0.5
470 Image Processing: The Fundamentals

i2 = i0 + α(t − 2) cos[β(t − 2)] + 0.5 j2 = j0 + α(t − 2) sin[β(t − 2)] + 0.5
...
iK = i0 + α(t − K) cos[β(t − K)] + 0.5 jK = j0 + α(t − K) sin[β(t − K)] + 0.5
(5.281)

In the above we must make sure that the values of the coordinates do not go out of
range, ie ix should take values between 1 and M and jx should take values between 1
and N . To ensure that, we use

ik = min{ik , M }
ik = max{ik , 1}
jk = min{jk , N }
jk = max{jk , 1}

for every k = 1, 2, . . . , K.
Step 6: Convert the coordinates computed in Step 5 into the indices of the column
vector we have created from the input image by writing its columns one under the
other. Given that a column has M elements indexed by i, with first value 1, the pixels
identified in (5.281) will have the following indices in the column image:

Ic = (ic − 1)M + jc
I1 = (i1 − 1)M + j1
I2 = (i2 − 1)M + j2
...
IK = (iK − 1)M + jK (5.282)

Step 7: If S(Ic ) = 0, we proceed to apply (5.284).

If S(Ic ) = 0, the elements of the Ic row of matrix H have already been computed. We
wish to retain, however, the most recent scrambling values, so we set them all again
to 0:
H(Ic , J) = 0 for all J = 1, 2, . . . , M N (5.283)
Then we proceed to apply (5.284):

S(Ic ) = 1
H(Ic , Ic ) = H(Ic , I1 ) = H(Ic , I2 ) = H(Ic , I3 ) = . . . = H(Ic , IK ) = 1 (5.284)

There will be some rows of H that have all their elements 0. This means that the
output pixel, that corresponds to such a row, will have value 0. We may decide to
allow this, in which case the scrambling we perform will not be easily invertible, as
matrix H will be singular. Alternatively, we may use the following fix.
Step 8: Check all rows of matrix H, and in a row that contains only 0s, set the
Inhomogeneous linear image restoration 471

diagonal element equal to 1. For example, if the 5th row contains only 0s, set the 5th
element of this row to 1. This means that the output pixel, that corresponds to this
row, will have the same value as the input pixel and matrix H will not be singular.
Step 9: Normalise each row of matrix H so that its elements sum up to 1.
After you have computed matrix H, you may produce the scrambled image g̃ in column
form, from the input image g, also in column form, by using:

g̃ = Hg (5.285)

Example 5.42
Figure 5.15a shows the image of a criminal that wishes to hide his face.
Use a window of 70 × 70 around his face to scramble it using the algorithm
of example 5.41.

In this case, M = N = 70. We select tmax = 50, 000, α = 0.001 and β = (2π/360)×2.
The value of α was chosen small so that the spiral is tight and, therefore, more likely
to pass through most, if not all, pixels. The value of β was selected to mean that
each time parameter t was incremented by 1, the spiral was rotated by 2o . The value
of t was selected high enough so that the spiral covers the whole square we wish to
scramble. After matrix H has been created and equation (5.285) applied to the 70 × 70
patch written as a 4900 vector, the result is wrapped again to form a 70 × 70 patch
which is embedded in the original image. The result is shown in figure 5.15b.

(a) (b)
Figure 5.15: (a) “Zoom” (size 360 × 256). (b) After a patch of size 70 × 70 around
the face region is scrambled.
472 Image Processing: The Fundamentals

Example B5.43

Instead of using the spiral of example 5.41 to scramble a subimage, use

concentric circles to scan a square subimage of size M × M .

Step 1: Create an array H of size M 2 × M 2 with all its elements 0.

Step 2: Choose (i0 , j0 ) to be the coordinates of a pixel near or at the centre of the
image.
Step 3: Create a 1D array S of M 2 samples, all with flag 0. This array will be used
to keep track which rows of matrix H have all their elements 0.
Step 4: Set β = (2π/360)x where x is a small number like 1 or 2. Select a value of
K, say K = 10.
Step 5: For α taking values from 1 to M/2 in steps of 1, do the following.
Step 6: Starting with t = 0 and carrying on with t = 1, 2, . . . , 359, perform the
following computations.
Compute the indices of the 2D image pixels that will have to be mixed to yield the
value of output pixel (ic , jc ):

ic = i0 + α cos(βt) + 0.5 jc = j0 + α sin(βt) + 0.5

i1 = i0 + α cos[β(t − 1)] + 0.5
i2 = i0 + α cos[β(t − 2)] + 0.5
...
iK = i0 + α cos[β(t − K)] + 0.5
j1 = j0 + α sin[β(t − 1)] + 0.5
j2 = j0 + α sin[β(t − 2)] + 0.5
jK = j0 + α sin[β(t − K)] + 0.5
(5.286)

In the above, we must make sure that the values of the coordinates do not go out of
range, ie ix and jx should take values between 1 and M . To ensure that, we use

ik = min{ik , M }
ik = max{ik , 1}
jk = min{jk , M }
jk = max{jk , 1}

for every k = 1, 2, . . . , K.
Step 7: Convert the coordinates computed in Step 5 into the indices of the column
vector we have created from the input image by writing its columns one under the
other. Given that a column has M elements indexed by i, with first value 1, the pixels
identified in (5.286) will have the following indices in the column image:
Inhomogeneous linear image restoration 473

Ic = (ic − 1)M + jc
I1 = (i1 − 1)M + j1
I2 = (i2 − 1)M + j2
...
IK = (iK − 1)M + jK (5.287)

Step 8: If S(Ic ) = 0, proceed to apply (5.289).

If S(Ic ) = 0, the elements of the Ic row of matrix H have already been computed. We
wish to retain, however, the most recent scrambling values, so we set them all again
to 0:
H(Ic , J) = 0 for all J = 1, 2, . . . , M 2 (5.288)
We then set:

S(Ic ) = 1
H(Ic , Ic ) = H(Ic , I1 ) = H(Ic , I2 ) = H(Ic , I3 ) = . . . = H(Ic , IK ) = 1 (5.289)

Step 9: Check all rows of matrix H, and in a row that contains only 0s, set the
diagonal element equal to 1.
Step 10: Normalise each row of matrix H so that its elements sum up to 1.

Example B5.44

Construct a scrambling matrix that averages the values of 2K + 1 pixels on

the circle of an arc centred at each pixel of an M × M patch of an image
and leaves no pixel unchanged.

Let us say that we wish to scramble a patch centred at pixel (i0 , j0 ). Consider a pixel
(i, j) of the patch. The polar coordinates of this pixel with respect to the patch centre
are !
r = (i − i0 )2 + (j − j0 )2 (5.290)
and θ, such that:
i = r cos θ and j = r sin θ (5.291)
Then points on the arc of the same circle centred at this pixel and symmetrically placed
on either side of it, have coordinates

ik = i0 + r cos(θ + kφ)
jk = j0 + r sin(θ + kφ) (5.292)
474 Image Processing: The Fundamentals

where k takes values −K, −K + 1, . . . , 0, . . . , K − 1, K and φ is the angle subtended

by a single pixel on the circle of radius r, with its vertex at the centre of the circle,
measured in rads: φ = 1/r.
Then the algorithm of creating matrix H is as follows.

Step 1: Create an array H of size M 2 × M 2 with all its elements 0.

Step 2: Choose (i0 , j0 ) to be the coordinates of a pixel near or at the centre

of the image.

Step 3: Scan every pixel (i, j) inside the subimage you wish to scramble, and
compute for it its polar coordinates using equations (5.290) and (5.291), and set
φ = 1/r.

Step 4: Compute the indices of the 2D image pixels that will have to be mixed to
yield the value of output pixel (i, j)
ik = i0 + r cos(θ + kφ) + 0.5 jk = j0 + r sin(θ + kφ) + 0.5 (5.293)
for k = −K, −K + 1, . . . , 0, . . . , K − 1, K.
In the above we must make sure that the values of the coordinates do not go out of
range, ie ik and jk should take values between 1 and M . To ensure that, we use
ik = min{ik , M }
ik = max{ik , 1}
jk = min{jk , M }
jk = max{jk , 1} (5.294)
for every k.

Step 5: Convert the coordinates computed in Step 4 into the indices of the
column vector we have created from the input image by writing its columns one under
the other. Given that a column has M elements indexed by i, with first value 1, the
pixels identified in (5.293) will have the following indices in the column image:
I = (i − 1)M + j
Ik = (ik − 1)M + jk (5.295)

Step 6: Set:
H(I, Ic ) = H(I, I−K ) = H(I, I−K+1 ) = . . . = H(I, I0 ) = . . . = H(I, IK ) = 1 (5.296)

Step 7: Check all rows of matrix H, and in a row that contains only 0s, set the
diagonal element equal to 1.

Step 8: Normalise each row of matrix H so that its elements sum up to 1.

Inhomogeneous linear image restoration 475

Example B5.45

Show how a thick whirl-like scrambling pattern might be created.

To keep it simple, when we compute a scrambled value for pixel (ic , jc ) we assign it
to all pixels around it inside a window of size (2L + 1) × (2L + 1). At first sight this
may appear to create image patches with the same value, but due to the continuous
and slow rotation of the spiral pattern, large parts of each square patch are continually
over-written and the effect disappears.

Example B5.46
Use the algorithms you developed in examples 5.43 and 5.45 to scramble
the face of figure 5.15a. Show the scrambled patterns and compare them
with the one produced in example 5.42.

(a) (b) (c)

(d) (e) (f )
Figure 5.16: (a) The original image to be scrambled (size 70 × 70). (b) The scrambling
obtained in example 5.42. (c) The scrambling obtained with the algorithm of example
5.43, with x = 1. (d) The scrambling obtained with the algorithm of example 5.43, with
x = 2. (e) The scrambling obtained with the algorithm of example 5.45, with α = 0.1,
x = 2, K = 50, L = 3 and tmax = 50, 000. (f ) The scrambling obtained with the
algorithm of example 5.45, with α = 0.03, x = 2, K = 50, L = 3 and tmax = 50, 000.
476 Image Processing: The Fundamentals

Example 5.47

Apply the algorithm of example 5.45 to construct matrix H with which an

8 × 8 image may be scrambled. Consider as the eye of the whirl pixel (4, 4).
Use this matrix then to scramble the flower image. Take the inverse of
matrix H and apply it to the scrambled image to reconstruct the flower.

As the image we have to scramble is small, only the inner part of the whirl will be used.
A large part of the whirl remains close to the central pixel, so although the image is
small, we have to use a large value of K. This is because K really represents the steps
along the whirl we use for averaging, not necessarily the number of distinct pixels, as
many of these steps are mapped to the same pixel. After trial and error, the following
parameters gave good results: α = 0.01, x = 1, K = 10, L = 1 and tmax = 5, 000.
Matrix H, of size 64 × 64, is shown in figure 5.17. Every black cell in this matrix
represents a nonzero value. The values along each row of the matrix are all equal and
sum up to 1. Every white cell represents a 0. We can see that there is no particular
structure in this matrix.

Figure 5.17: Matrix H with which an 8 × 8 image may be scrambled. All black cells
represent nonzero positive numbers that along each row are equal and sum up to 1.
White cells represent value 0.
Inhomogeneous linear image restoration 477

Figure 5.18 shows the original image, the scrambled one and the unscrambled obtained
by operating on the scrambled image with matrix H −1 . The sum of the squares of
the errors of the reconstructed image is 219. This error is computed from the raw
output values, where negative and higher than 255 grey values are allowed. This error
is only due to the quantisation errors introduced by representing the scrambled image
with integers, as matrix H could be inverted exactly. We observe that although we
knew exactly what the scrambling matrix was and we applied its inverse exactly on
the scrambled image, we did not get back the flower image exactly. This is because in
equation (5.285), on page 471, the g̃ is given to us as an image with all its elements
rounded to the nearest integer, while the result of Hg is actually a vector with non-
integer elements. So, the application of H −1 to g̃ is not the inverse of equation (5.285).

(a) (b) (c) (d)

Figure 5.18: (a) Original image of size 8 × 8. (b) Image scrambled with matrix H
shown in figure 5.17. (c) Image recovered by operating on the scrambled image with
the inverse of matrix H. The output has all its values beyond 255 and below 0 truncated
to 255 and 0, respectively. The square error is 79. (d) Image recovered by operating
on the scrambled image with the inverse of matrix H. The output values are mapped
to the range 0 to 255, without any truncation. The square error is 39957.

How may we use constrained matrix inversion when the distortion matrix is not
circulant?

Here the regularisation term has to be applied to the whole matrix, and not in the form of
a filter. To do that, we have to solve the following problem: if g̃ is the observed distorted
image, g is the undistorted image we are seeking and H is the distortion matrix that was
used to produce g̃, minimise Lg and H −1 g̃ − g simultaneously. Here L is a matrix similar
to the one we worked out in example 5.30, on page 449. If it operates on a column image, it
yields the Laplacian at each position of the image. However, other regularisation constraints
may also be used (see examples 5.48 and 5.49). We formulate the problem as one in which
the norm of vectors Lg and H −1 g̃ − g have to be minimised simultaneously, and express that
by a so called cost function that has to be minimised:

U (g) ≡ ||H −1 g̃ − g||2 + λ||Lg||2 (5.297)

478 Image Processing: The Fundamentals

Here parameter λ allows us to control how much smoothing we want to impose to the data.
Our problem now is to specify values for g so that U (g) is minimum. Note that U (g) is
quadratic with respect to all the unknowns (the elements of vector g), so this is a classical
minimisation problem: we must find the first derivatives of U (g) with respect to the unknowns,
set them to 0 and solve the linear system of equations that will result from that. So, our first
task is to write U (g) explicitly in terms of the components of vector g and then work out its
derivatives with respect to these components.
For simplicity, let us call H −1 g̃ ≡ d. Vector d is the noisy estimate we have for the
elements of the original image. Now we want to select values gi which will be close to di and
at the same time they will change smoothly from one pixel position to the next. Assuming
that the image is M × N , U (g) may be written as:
⎡ ⎤2
MN MN MN
U (g) ≡ (di − gi )2 + λ ⎣ Lij gj ⎦ (5.298)
i=1 i=1 j=1

We have to set the first derivative of U (g) with respect to one of the components of g, say
gk , equal to 0:
∂U
= 0
∂gk
⎡ ⎤
MN MN
⇒ 2(dk − gk )(−1) + λ2 ⎣ Lij gj ⎦ Lik = 0
i=1 j=1
⎡ ⎤
MN MN
⇒ gk + λ ⎣ Lij ⎦ Lik gj = dk (5.299)
i=1 j=1

Example 5.48

A column image consisting of 3 elements has been scrambled by a 3 × 3

matrix, the inverse of which is matrix A. We wish to restore the image,
using as a regularisation constraint the requirement that every pixel has a
value as similar as possible with the value of the next pixel. Work out the
system of equations you will have to solve in this case.

Let us call the scrambled image we have g̃. Operating on it with matrix A will yield a
noisy estimate of the original image. Let us call it d:

d1 = a11 g̃1 + a12 g̃2 + a13 g̃3

d2 = a21 g̃1 + a22 g̃2 + a23 g̃3
d3 = a31 g̃1 + a32 g̃2 + a33 g̃3 (5.300)

We wish to estimate better values for the original image g which will be such that the
squares of first differences g1 − g2 , g2 − g3 and g3 − g1 are as small as possible, and at
Inhomogeneous linear image restoration 479

the same time gi will be as near as possible to di . Note that we assumed here periodic
boundary conditions, ie that the image is repeated ad infinitum, so that the last pixel g3
has as its next neighbour the first pixel g1 . Then the cost function we have to minimise
is:

U = (d1 −g1 )2 +(d2 −g2 )2 +(d3 −g3 )2 +λ(g1 −g2 )2 +λ(g2 −g3 )2 +λ(g3 −g1 )2 (5.301)

Parameter λ is used to control the balance between remaining faithful to the original
estimate and imposing the smoothness constraint we selected to use. The derivatives
of this function with respect to the three unknowns are:
∂U
= 2(d1 − g1 )(−1) + 2λ(g1 − g2 ) − 2λ(g3 − g1 )
∂g1
∂U
= 2(d2 − g2 )(−1) − 2λ(g1 − g2 ) + 2λ(g2 − g3 )
∂g2
∂U
= 2(d3 − g3 )(−1) − 2λ(g2 − g3 ) + 2λ(g3 − g1 ) (5.302)
∂g3
The right-hand sides of these equations should be set to 0. Then, after some manipu-
lation, the system of equations we have to solve becomes:

g1 (1 + 2λ) − λg2 − λg3 = d1

−λg1 + g2 (1 + 2λ) − λg3 = d2
−λg1 − λg2 + (1 + 2λ)g3 = d3 (5.303)

Example 5.49

Verify that the result of example 5.48 could have been obtained by applying
formula (5.299).

We first work out matrix L, which, when it operates on the image, yields at each
position the first difference with the next pixel. It is easy to see that:
⎛ ⎞ ⎛ ⎞⎛ ⎞
g1 − g2 1 −1 0 g1
⎝ g2 − g3 ⎠ = ⎝ 0 1 −1 ⎠ ⎝ g2 ⎠ (5.304)
g3 − g1 −1 0 1 g3

So, matrix L is:

⎛ ⎞
1 −1 0
⎝ 0 1 −1 ⎠ (5.305)
−1 0 1
480 Image Processing: The Fundamentals

For M N = 3 and for k = 1, 2, 3, formula (5.299) has the form:

3 3
g1 + λ Lij Li1 gj = d1
i=1 j=1
3 3
g2 + λ Lij Li2 gj = d2
i=1 j=1
3 3
g3 + λ Lij Li3 gj = d3 (5.306)
i=1 j=1

Or:
3
g1 + λ (Li1 Li1 g1 + Li2 Li1 g2 + Li3 Li1 g3 ) = d1
i=1
3
g2 + λ (Li1 Li2 g1 + Li2 Li2 g2 + Li3 Li2 g3 ) = d2
i=1
3
g3 + λ (Li1 Li3 g1 + Li2 Li3 g2 + Li3 Li3 g3 ) = d3 (5.307)
i=1

Or:

g1 + λ(L11 L11 g1 + L12 L11 g2 + L13 L11 g3

+L21 L21 g1 + L22 L21 g2 + L23 L21 g3
+L31 L31 g1 + L32 L31 g2 + L33 L31 g3 ) = d1

g2 + λ(L11 L12 g1 + L12 L12 g2 + L13 L12 g3

+L21 L22 g1 + L22 L22 g2 + L23 L22 g3
+L31 L32 g1 + L32 L32 g2 + L33 L32 g3 ) = d2

g3 + λ(L11 L13 g1 + L12 L13 g2 + L13 L13 g3

+L21 L23 g1 + L22 L23 g2 + L23 L23 g3
+L31 L33 g1 + L32 L33 g2 + L33 L33 g3 ) = d3 (5.308)

Substituting the values of Lij , we finally get:

g1 + λ(g1 − g2 + g1 − g3 ) = d1
g2 + λ(−g1 + g2 + g2 − g3 ) = d2
g3 + λ(−g2 + g3 − g1 + g3 ) = d3 (5.309)

This set of equations is the same as (5.303).

Inhomogeneous linear image restoration 481

What happens if matrix H is really very big and we cannot take its inverse?

In such cases, instead of estimating d ≡ H −1 g̃ and trying to estimate g so that it is as close

as possible to d, we try to estimate g so that Hg is as close as possible to g̃. The function
we have to minimise now has the form:
U (g) ≡ ||g̃ − Hg||2 + λ||Lg||2 (5.310)
In terms of components, U (g) may be written as:
⎛ ⎞2 ⎡ ⎤2
MN MN MN MN
U (g) ≡ ⎝g̃i − Hij gj ⎠ + λ ⎣ Lij gj ⎦ (5.311)
i=1 j=1 i=1 j=1

We have to work out the first derivative of U (g) with respect to a component of g, say gk ,
and set it equal to 0:
∂U
= 0
∂gk
⎡ ⎤
MN MN MN MN
⇒2 ⎣g̃i − Hij gj ⎦ (−Hik ) + 2λ Lij gj Lik = 0
i=1 j=1 i=1 j=1
MN MN MN MN MN
⇒− g̃i Hik + Hij Hik gj + λ Lij Lik gj = 0
i=1 i=1 j=1 i=1 j=1
MN
3M N 4 MN
3 MN
4 MN
⇒ gj Hij Hik + gj λ Lij Lik = Hik g̃i
j=1 i=1 j=1 i=1 i=1
MN MN MN
⇒ gj (Hij Hik + λLij Lik ) = Hik g̃i
j=1 i=1 i=1
     
≡Akj ≡bk
MN
⇒ Akj gj = bk
j=1
⇒ Ag = b (5.312)
Matrix A ≡ H T H + λLT L with elements
MN
Akj ≡ (Hik Hij + λLik Lij ) (5.313)
i=1

and vector b ≡ H T g̃ with elements

MN
bk ≡ Hik g̃i (5.314)
i=1

may easily be computed from the distorted image g̃, the distortion matrix H and the regu-
larisation matrix L.
482 Image Processing: The Fundamentals

However, system (5.312) is still very difficult to solve, because matrix A cannot be inverted
easily. It is neither circulant nor block circulant and it is of size M N × M N , where M N
typically could be of the order of 250, 000. Approximate iterative methods may be used
in such a case, like Jacobi’s method and its more elaborate version, the Gauss-Seidel
method.

Box 5.7. Jacobi’s method for inverting large systems of linear equations

Consider the system of N linear equations with N unknowns

Ax = b (5.315)

where A is an N × N known matrix and b is an N × 1 known vector. If matrix A has

its largest values along its diagonal, then system (5.315) may be solved iteratively, as
follows.

Step 1: Write matrix A as the sum of three matrices, one containing its ele-
ments along its diagonal, one containing its elements in the upper triangle of it, and
the other containing its elements in the lower triangle of it:
⎛ ⎞ ⎛ ⎞ ⎛ ⎞
a11 0 . . . 0 0 a12 . . . a1N 0 0 ... 0
⎜ 0 a22 . . . 0 ⎟ ⎜ 0 0 . . . a2N ⎟ ⎜ a21 0 ... 0 ⎟
⎜ ⎟ ⎜ ⎟ ⎜ ⎟
D≡⎜ . .. .. ⎟ U ≡ ⎜ .. .. .. ⎟ L ≡ ⎜ .. .. .. ⎟
⎝ .. . ... . ⎠ ⎝ . . ... . ⎠ ⎝ . . ... . ⎠
0 0 . . . aN N 0 0 ... 0aN 1 aN 2 . . . 0
(5.316)
Step 2: Make an initial guess for the values of the elements of vector x. Say, set them
all to 0.
Step 3: Update the values of x until convergence, using the iterative formula

Dx(k+1) = −(L + U )x(k) + b (5.317)

where superscript (k) means the value at iteration k.

Note that in practice we do not use matrices, as the computer may not be able to
handle such large matrices: we instead consider each equation in turn, as follows.

Step 2 (realistic): Initialise all elements of x:

(0) bi
xi = for i = 1, 2, . . . , N (5.318)
Dii
Step 3 (realistic): Update the values of xi , iteratively, until convergence:
⎛ ⎞
N
1 ⎝
Aij xj ⎠
(k+1) (k)
xi = bi − for i = 1, 2, . . . , N (5.319)
Dii
j=1,j=i
Inhomogeneous linear image restoration 483

The essence of this algorithm is to consider only the dominant term in each equation,
assuming that this is the first term for the first equation, the second term for the second
equation, the third term for the third equation, and so on, and express the unknown
of the dominant term in terms of the other unknowns. Then compute an improved
value of the dominant unknown in each equation from the estimated values of the other
unknowns (see example 5.50). Convergence is guaranteed when the absolute value of
the diagonal element in each row of matrix A is larger than the sum of the absolute
values of the other elements.

Example B5.50

Consider the system of equations Ax = b, or explicitly

a11 x1 + a12 x2 + a13 x3 = b1

a21 x1 + a22 x2 + a23 x3 = b2
a31 x1 + a32 x2 + a33 x3 = b3 (5.320)

where |a11 | > |a12 | + |a13 |, |a22 | > |a21 | + |a23 | and |a33 | > |a31 | + |a32 |. Solve it
for x1 , x2 and x3 , iteratively.

First we keep the dominant term in each equation, and express it in terms of the other
terms:

a11 x1 = b1 − a12 x2 − a13 x3 = b1 − 0x1 − a12 x2 − a13 x3

a22 x2 = b2 − a21 x1 − a23 x3 = b2 − a21 x1 − 0x2 − a23 x3
a33 x3 = b3 − a31 x1 − a32 x2 = b3 − a31 x1 − a32 x2 − 0x3 (5.321)

In terms of matrices, this system of equations may be written as:

⎛ ⎞⎛ ⎞ ⎛ ⎞ ⎛ ⎞⎛ ⎞
a11 0 0 x1 b1 0 a12 a13 x1
⎝ 0 a22 0 ⎠ ⎝ x2 ⎠ = ⎝ b2 ⎠ − ⎝ a21 0 a23 ⎠ ⎝ x2 ⎠ (5.322)
0 0 a33 x3 b3 a31 a32 0 x3

Note that the matrix that appears on the left-hand side is the matrix of the diagonal
elements of matrix A, while the matrix that appears on the right-hand side is the sum of
the upper and the lower triangular matrices of matrix A. The values of the unknowns
on the right-hand side are to be the values we have estimated so far, while the values
we shall compute this way will be improved estimates. This way of solving the problem
is formally given in Box 5.7 as Jacobi’s method.
484 Image Processing: The Fundamentals

Example B5.51

In example 5.50 observe that when you solve system (5.321), at each iter-
ation you find first an improved value for x1 , then for x2 , and finally for x3 .
Use the improved value of x1 when you work out the new estimate for x2 ,
and the improved values of x1 and x2 when you work out the new estimate
for x3 .

Let us call the improved values we work out at an iteration (ξ1 , ξ2 , ξ3 )T . The system
(5.321) may be written as:

a11 ξ1 = b1 − 0x1 − a12 x2 − a13 x3

a22 ξ2 = b2 − a21 x1 − 0x2 − a23 x3
a33 ξ3 = b3 − a31 x1 − a32 x2 − 0x3 (5.323)

Now consider that when you compute ξ2 , instead of using on the right-hand side the
old value of x1 , you use the newly estimated value ξ1 , and when you work out ξ3 , you
make use of the values ξ1 and ξ2 , instead of x1 and x2 . Then this system becomes:

a11 ξ1 = b1 − 0x1 − a12 x2 − a13 x3

a22 ξ2 = b2 − a21 ξ1 − 0x2 − a23 x3
a33 ξ3 = b3 − a31 ξ1 − a32 ξ2 − 0x3 (5.324)

In terms of matrices, this may be written as:

⎛ ⎞⎛ ⎞ ⎛ ⎞ ⎛ ⎞⎛ ⎞
a11 0 0 ξ1 b1 0 0 0 ξ1
⎝ 0 a22 0 ⎠ ⎝ ξ2 ⎠ = ⎝ b2 ⎠ − ⎝ a21 0 0 ⎠ ⎝ ξ2 ⎠
0 0 a33 ξ3 b3 a31 a32 0 ξ3
⎛ ⎞⎛ ⎞
0 a12 a13 x1
−⎝ 0 0 a23 ⎠ ⎝ x2 ⎠ (5.325)
0 0 0 x3

Collecting all the new estimates on the left-hand side, we eventually get:
⎧⎛ ⎞ ⎛ ⎞⎫ ⎛ ⎞
⎨ a11 0 0 0 0 0 ⎬ ξ1
⎝ 0 a22 0 ⎠ + ⎝ a21 0 0 ⎠ ⎝ ξ2 ⎠ =
⎩ ⎭
0 0 a33 a31 a32 0 ξ3
⎛ ⎞ ⎛ ⎞⎛ ⎞
b1 0 a12 a13 x1
⎝ b2 ⎠ − ⎝ 0 0 a23 ⎠ ⎝ x2 ⎠ (5.326)
b3 0 0 0 x3

This formulation of the problem is known as the Gauss-Seidel method (see Box
5.8).
Inhomogeneous linear image restoration 485

Box 5.8. Gauss-Seidel method for inverting large systems of linear equations

Note that in Jacobi’s method (Box 5.7) we may solve the system sequentially, using for
some of the unknowns the improved values we have already estimated in the current
iteration, instead of waiting to update all values using those of the previous iteration
(see example 5.51). In that case, the algorithm is like that in Box 5.7, but with step 3
replaced with:
Step 3 (Gauss-Seidel): Update the values of x until convergence, using the iterative
formula
(D + L)x(k+1) = −U x(k) + b (5.327)
where superscript (k) means the value at iteration k.
Again, this step in practice is performed by handling each equation separately:

Step 3 (Gauss-Seidel, realistic): Update the values of xi until convergence,

using:
⎛ ⎞
N N
(k+1) 1 ⎝bi − (k) (k+1) ⎠
xi = Aij xj − Aij xj for i = 1, 2, . . . , N
Dii j=1,j>i j=1,j<i
(5.328)

Does matrix H as constructed in examples 5.41, 5.43, 5.44 and 5.45 fulfil the
conditions for using the Gauss-Seidel or the Jacobi method?

Only marginally. The diagonal elements of matrix H are not larger than the non-diagonal
elements. We may, however, construct scrambling matrices H which fulfil this constraint, if
we add the following step at the end of each algorithm:

Step easy: Count the nonzero elements of each row of matrix H. If it is only 1, do nothing.
If there are N > 1 nonzero elements, set the value of the diagonal element to 0.55. Set the
values of the other nonzero elements to 0.45/(N − 1).

Example 5.52

Degrade the face of image 5.15a with the algorithm of example 5.45 with
the extra step that makes the scrambling matrix dominated by its diagonal
values. Then unscramble it using the Jacobi and the Gauss-Seidel methods.

Figure 5.19a shows the original image 5.16a. Its degraded version, shown in 5.19b,
was obtained by running the thick whirl algorithm with parameters α = 0.01, x = 2,
K = 25, L = 1 and using 5000 steps, and subsequently modifying the scrambling
486 Image Processing: The Fundamentals

matrix H so that the dominant element of each row is the diagonal element, with a
value larger than the sum of the values of the remaining elements. This is necessary
for the restoration to be possible with the Jacobi or the Gauss-Seidel method. To
achieve this, we set all diagonal elements of the matrix to 0.55 (unless the diagonal
element was the only nonzero element in its row, in which case it was left to be 1),
and distributed the remaining 0.45 (necessary to make all elements of a row to sum
up to 1) equally between the remaining nonzero elements of the same row. The H
matrix created this way is not really effective in scrambling, as the face can be easily
recognised. Nevertheless, the face is degraded, and our task is to recover its undegraded
version.
Figures 5.19c and 5.19d show the recovered images by using the Jacobi and the Gauss-
Seidel algorithm, respectively. The algorithms were run with the following stopping
criterion: “stop either when the number of iterations reaches 35, or when the sum of
the squares of the differences in the values of the pixels between successive iterations is
less than 0.1”. Note that the values of the pixels were in the range [0, 1] and not [0, 255].
The Jacobi algorithm stopped after 29 iterations with a sum square error 0.09, while
the Gauss-Seidel algorithm stopped after 11 iterations with sum square error 0.011.
Although one cannot see any difference in the quality of the results, the Gauss-Seidel
algorithm converged faster, to a more accurate solution.

(a) Original (b) Degraded (c) Restored (Jac) (d) Restored (G-S)
M SE = 606 M SE = 0.2425 M SE = 0.0066

Figure 5.19: An original image, its degraded version and the recovered images by the
Jacobi and the Gauss-Seidel algorithms. The mean square errors were computed using
pixel values in the range [0, 255].

What happens if matrix H does not satisfy the conditions for the Gauss-Seidel
method?

In that case we can minimise the cost function (5.297) by using gradient descent. Since the
function is quadratic in all the unknowns, it has a well defined minimum that can be reached
by simply starting from an initial guess of the solution and then updating the values of the
unknowns, one at a time, so that we always move towards the minimum. This is shown
schematically in figure 5.20.
Inhomogeneous linear image restoration 487

U(x)

A
B

x
x1 x0 x2

Figure 5.20: A function U (x), which is quadratic in its argument, has one well defined
minimum at point C, for x = x0 . If we guess an initial value for x0 as x = x1 , we are
at point A, where, for increasing value of x, the function decreases, ie dU/dx < 0. Then if
we increase the value of the original guess, we shall move it towards x0 . If we guess an initial
value for x0 as x = x2 , we are at point B, where, for increasing value of x, the function also
increases, ie dU/dx > 0. Then if we decrease the value of the original guess, we shall move it
towards x0 . In either case, we change the value of the original guess by −dU/dx.

How do we apply the gradient descent algorithm in practice?

Step 1: Guess an initial restored image. You may set it to be the same as the distorted
image, or equal to a flat image where all pixels have value 128.

Step 2: Consider in turn, one pixel at a time. Compute the value of the first derivative
of function (5.310) with respect to the pixel value under consideration
MN
∂U
= gj Akj − bk (5.329)
gk j=1

where parameters Akj and bk are given by (5.313) and (5.314), respectively.

∂U
Step 3: If gk > 0, decrease the value of pixel k by 1.

∂U
If gk < 0, increase the value of pixel k by 1.

∂U
If gk = 0, do nothing.

You may set as a termination criterion a fixed number of updates. For an image with M N
pixels, and since we change the value of each pixel by 1 at each update, and if we start with
all pixels having value 128, we may estimate that we shall need roughly 128M N updates, ie
128 iterations.