2/20/24

Computer Vision
Course
School of Information and
Communication Technology

1
 2/20/24

Computer Vision
Chapter 3. Image Processing
(Part 2)
Computer Vision Group
School of Information and Communications Technology

Contents

• Remind: digital image

representation
• Point Operators
• Convolution and spatial
filtering
• More neighborhood operators
• Image transforms

CV Group – School of Information and Communication and Technology

4
4

2
 2/20/24

Arithmetical/Logical Operations

• AND operation
• OR operation
• Image subtraction
• Image addition

AND operation

Original And mask Output image

3
 2/20/24

OR operation

Original OR mask Output image

Image Addition

• If f and g are two images, the

pixelwise addition R is defined as:
• R(x,y) = Min( f(x,y)+g(x,y) ; 255 )

• Image addition is used to

• lower the noise in a serie of images
• increase the luminance by adding the
image to itself

Source : Eric Favier. L'analyse et le traitement des images. ENISE.

4
 2/20/24

Average Images

• g(x,y) is the addition of f(x,y) and noise 𝜂(𝑥, 𝑦)

• If we have several images {g(x,y)}, we can compute

the average one

Average Images - Examples

10

10

5
 2/20/24

Image subtraction

• The pixelwise substraction

of two images f and g is:
S(x,y) = Max( f(x,y)-g(x,y) ; 0 )

• Image substraction is used

to
• detect defaults, detect
difference between images
• detect motion in images

Source : Eric Favier. L'analyse et le traitement des images. ENISE.

11

11

Image subtraction

12

12

6
 2/20/24

Image subtraction

After detection, we still have some noise, that

we can clean to keep only the object of interest

13

13

Image multiplication

• The multiplication S of an image f by a ratio (factor)

is defined as:
• S(x,y) = Max( f(x,y)*ratio ; 255)
• Image multiplication can be used to increase the
contrast or the luminosity

Source : Eric Favier. L'analyse et le traitement des images. ENISE.

14

14

7
 2/20/24

Operations on images

Source : www.nte.montaigne.u-bordeaux.fr/SuppCours/5314/Dai/TraitImage01-02.ppt
15

15

Exercise

• Given two images as bellow

1. Transform images to negative ones

2. Process to have an image which has only the “ball”

16

16

8
 2/20/24

Binary images

18

Binary images
• Two pixel values: foreground (object, 1) and
background (0)
• Be used
• To mark region(s) of interest
• As results of thresholding method

19

9
 2/20/24

Thresholding
• Given a grayscale image or an intermediate matrix à
threshold to create a binary output.

Example: edge detection

Gradient magnitude fg_pix = find(gradient_mag > t);

Looking for pixels where gradient is strong.

20

Thresholding
• Given a grayscale image or an intermediate matrix à
threshold to create a binary output.
Example: background subtraction

- =

Looking for pixels that differ

significantly from the “empty”
background.
fg_pix = find(diff > t);

21

10
 2/20/24

Thresholding
• Given a grayscale image or an intermediate matrix à
threshold to create a binary output.

Example: intensity-based detection

fg_pix = find(im < 65);

Looking for dark pixels

22

Thresholding
• Given a grayscale image or an intermediate matrix à
threshold to create a binary output.
Example: color-based detection

fg_pix = find(hue > t1 & hue < t2);

Looking for pixels within a certain hue range.

Slide credit: Kristen Grauman

23

11
 2/20/24

Issues

• What to do with “noisy” binary

outputs?
• Holes
• Extra small fragments

• How to demarcate multiple regions

of interest?
• Count objects
• Compute further features per object

Slide credit: Kristen Grauman

24

Morphological operators

• Change the shape of the foreground regions via

intersection/union operations between a scanning
structuring element and binary image.
• Useful to clean up result from thresholding

• Main components
• Structuring element
• Operators:
• Basic operators: Dilation, Erosion
• Others: Opening, Closing, …

25

12
 2/20/24

Dilation

• Expands connected components

• Grow features
• Fill holes

Before dilation After dilation

Slide credit: Kristen Grauman

26

Erosion

• Erode connected components

• Shrink features
• Remove bridges, branches, noise

Before erosion After erosion

Slide credit: Kristen Grauman

27

13
 2/20/24

Structuring elements

• Masks of varying shapes and sizes used to perform

morphology, for example:

• Scan mask (structuring element) over the object

(foreground) borders (inside and outside) and
transform the binary image

28

Dilation

• Moving S on each pixel of A

• check if the intersection (pixels belonging to object) is not empty
• If yes, the center of B belongs to the result image
• If a pixel of S is onto object pixels (A), then the central pixel belongs to object
• Otherwise (i.e. all pixels of are background), set to background (no change)

𝑨 ⊕S

Structuring element
S

29

14
 2/20/24

Dilation

• As max filter
• Can be applied both on
• binary images
• or grayscale images

Mask /
Structuring element

30

Erosion

• We put the element S on each pixel x of A

• like convolution

• If all pixels of S are onto object pixels (A), then the central pixel belongs to
object
• Otherwise (i.e. a mask pixel is background), set to background

𝑨 ⊝S

Structuring
element
S

31

15
 2/20/24

Erosion

• As min filter
• Can be applied both on
• binary images
• or grayscale images

Mask /
Structuring element

32

2D example

Shapiro & Stockman

33

16
 2/20/24

Opening

• Erode, then dilate

• Remove small objects, keep original shape

Before opening After opening

34

Closing

• Dilate, then erode

• Fill holes, but keep original shape

Before closing After closing

demo: http://bigwww.epfl.ch/demo/jmorpho/start.php

35

17
 2/20/24

Opening vs Closing

Structuring element

Opening

Closing

36

Connected component labeling

• We loop over all the image to give a unique number (label) for each region
• All pixels from the same region must have the same number (label)
• Objectifs:
• Counting objects
• Separating objets
• Creating a mask for each object
• …

37

18
 2/20/24

Connected component labeling

First loop over the image

• For each pixel in a region, we set
• or the smallest label from its top or left neighbors
• or a new label

38

Connected component labeling

First loop over the image

• For each pixel in a region, we set
• or the smallest label from its top or left neighbors
• or a new label

39

19
 2/20/24

Connected component labeling

First loop over the image

• For each pixel in a region, we set
• or the smallest label from its top or left neighbors
• or a new label

40

Connected component labeling

First loop over the image

• For each pixel in a region, we set
• or the smallest label from its top or left neighbors
• or a new label

41

20
 2/20/24

Connected component labeling

Second loop over the image

• For each pixel in a region, we set
• the smallest from its own label and the labels from its down and right neighbors

42

Connected component labeling

Second loop over the image

• For each pixel in a region, we set
• the smallest from its own label and the labels from its down and right neighbors

43

21
 2/20/24

Connected component labeling

Second loop over the image

• For each pixel in a region, we set
• the smallest from its own label and the labels from its down and right neighbors

44

Connected component labeling

• Two loops are enough?

• example: spiral region !

• Solutions
• We continue, go and back two ways, until no
new change in labels

• It is possible to do only one loop: manage a

table of equivalences when 2 different labels
are neighbors

45

22
 2/20/24

CC labeling: how many neighbors?

• Advice: Use different connexities for edges and regions

• 4-Connexity for regions
• 8-Connexity for edges

46

CC labeling: how many neighbors?

• Regions labeling
• We use 4-connexity
• Each loop, we compare 2 neighbors

• Edge labeling
• 8-connexity
• Each loop, we compare 4 neighbors

47

23
 2/20/24

Contents

• Remind: digital image

representation
• Point Operators
• Convolution and spatial
filtering
• More neighborhood operators
• Image transforms
• Frequency domain
• PCA

CV Group – School of Information and Communication and Technology

48
48

Frequencies in images

Intensity profile

49

49

24
 2/20/24

Frequencies in images

• What are the (low/high) frequencies in an image?

• Frequency = intensity change
• Slow changes (homogeneous /blur regions): low
frequency
• fast/abrupt changes (egde, contour, noise): high frequency

High frequency Most of

energy
concentrated
in low
Low frequency
frequencies

50

50

Low frequencies

51

51

25
 2/20/24

High frequencies

52

52

Image spectral analysis

• An image is a visual signal

• We can analyse the frequencies of the signal
• How?
• We will create a new « image » which will contains all
frequencies of the image
• Like a 2D frequency graphic
• The basic tool for it is the Fourier Transform

• We talk about the frequency domain, opposing to

the spatial domain (image)

53

53

26
 2/20/24

Frequencies in a signal

Source : Gonzalez and Woods. Digital Image Processing. Prentice-Hall, 2002.

54

54

Fourier series

A bold idea (1807) - Jean Baptiste

Joseph Fourier (1768-1830):

Any univariate function can be

rewritten as a weighted sum of
sines and cosines of different
frequencies.

Our building block:

A sin(wt ) + B cos(wt )
Add enough of them to get
any signal g(t) you want!

Hays

55

27
 2/20/24

Example

g(t) = sin(2πf t) + (1/3)sin(2π(3f) t)

t = [0,2], f = 1

g = +

t
Coefficient

Slides: Efros
56

56

Fourier Transform

• Fourier transform is a mathematical transform that

• Decomposes functions depending on space or time into
functions depending on spatial or temporal frequency

f(x) |F(𝟂)|
signal Fourier transform
of signal

57

57

28
 2/20/24

Fourier Series

We want to get this

function

Slide by Alexei A. Efros

58

Fourier Series

≈ +

We want to get this

function

Slide by Alexei A. Efros

59

29
 2/20/24

Fourier Series

≈ +

We want to get this

function

Slide by Alexei A. Efros

60

Fourier Series

≈ +

We want to get this

function

Slide by Alexei A. Efros

61

30
 2/20/24

Fourier Series

≈ +

We want to get this

function

Slide by Alexei A. Efros

62

Fourier Series

≈ +

We want to get this

function

Slide by Alexei A. Efros

63

31
 2/20/24

Fourier Series

We want to get this function We’ll get there in the limit

Slide by Alexei A. Efros

64

The math

• Where are the sines and cosines?

• The result is a complex function
• We’ve been showing only the amplitude A (spectre) so far:
• Phase is also encoded:

Slide by Steve Seitz

65

32
 2/20/24

Magnitude and phase

• Fourier transform stores the magnitude and phase at each

frequency
• Magnitude encodes how much signal there is at a particular frequency
• Phase encodes spatial information (indirectly)
• For mathematical convenience, this is often notated in terms of real
and complex numbers

Amplitude: Phase:
I (w )
A = ± R(w ) 2 + I (w ) 2
f = tan -1
R(w )

Slide by Rober Pless

66

Discrete Fourier transform

67

33
 2/20/24

Basic Fourier Transform pairs

Source: Václav Hlavác - Fourier transform, in 1D and in 2D

68

68

Basic Fourier Transform pairs

Source: Václav Hlavác - Fourier transform, in 1D and in 2D

69

69

34
 2/20/24

Basic Fourier Transform pairs

Source: Václav Hlavác - Fourier transform, in 1D and in 2D

70

70

Example: Music

• We think of music in terms of frequencies at different

magnitudes

Slide: Hoiem

71

35
 2/20/24

2D FFT

• Continuous FFT:

• Inverse FFT:

72

2D FFT - discrete

73

36
 2/20/24

Function cosin 2D

74

74

Separability of 2D DFT

• Notice that Fourier transform “filter elements” can

be expressed as products

2D DFT 1D DFT (row) 1D DFT (column)

• Formula above can be broken down into simpler

formulas for 1D DFT

75

75

37
 2/20/24

Separabilty of 2D DFT

• Using their separability property, can use 1D DFTs

to calculate rows then columns of 2D Fourier
Transform

(a) Original image (b) DFT of each row of a (c) DFT of each column of b
f(x,y) F1(x,v) F(u,v)

76

76

Properties of 2D DFT

• Linearity: DFT of a sum is equal to sum (or

multiplication) of the individual DFT’s
• Cyclicity (frequencies)
• Cycle?
• F(u,v) = F(u+a*M,v+b*N) a,b are integers
$%& (%&
𝑥𝑢 𝑦𝑣
𝐹(𝑢 + 𝑎 ∗ 𝑀, 𝑣 + 𝑏 ∗ 𝑁) = / / 𝑓 𝑥, 𝑦 ∗ 𝑒𝑥𝑝 −2𝜋𝑖( +𝑎+ + 𝑏)
𝑀 𝑁
!"# '"#
$%& (%&
𝑥𝑢 𝑦𝑣
= / / 𝑓 𝑥, 𝑦 ∗ 𝑒𝑥𝑝 −2𝜋𝑖( + ) ∗ 𝑒𝑥𝑝 −2𝜋𝑖(𝑎𝑥 + 𝑏𝑦)
𝑀 𝑁
!"# '"#

𝑒𝑥𝑝 −2𝜋𝑖(𝑎𝑥 + 𝑏𝑦) = cos 2𝜋 𝑎𝑥 + 𝑏𝑦 + sin 2𝜋 𝑎𝑥 + 𝑏𝑦 =1

77

77

38
 2/20/24

DC Component

• Recall that

• The value F(0,0) of DFT is called DC coefficient

• If we put u=v=0 then

• Essentially F(0,0) is the sum of all terms in the

original matrix

78

78

Image Fourrier transform

Enhanced Spectra
Original image Spectra | F(u,v) |
log (1+ | F(u,v) |)

Frequency
domain
Spatial
domain

79

79

39
 2/20/24

Image Fourrier transform

F(u,v) = R(u,v) + i*I(u,v)

Real part

Original image

Complex part

80

80

Image Fourrier transform

Point (0,0)

Why could we do it?

Switch(0,0) to the center of the image

81

81

40
 2/20/24

Image Fourrier transform

I II Point (0,0)
(0,0) (-M/2 , -N/2 )

IV III

(M-1, N-1) (M/2, N/2)

Frequency (M,N) of the Fourier transform

82

82

Image reconstruction from frequency images

F(u,v) = R(u,v) + i*I(u,v)

Real part

Original image

Complex part

83

83

41
 2/20/24

Convolution using 2D DFT

• DFT allows “convolution” by a different method

• “Convolution” becomes mutiplication

84

84

Fourier analysis in images

Intensity images (spatial domain)

Fourier images (spectral images – amtitude images)

http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
More: http://www.cs.unm.edu/~brayer/vision/fourier.html

85

85

42
 2/20/24

Signals can be composed

Intensity images (spatial domain)

+ =

Fourier images (spectral images – amtitude images)

http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering

86

86

Fourier Transform of an image

Fourier decomposition
Natural image Frequency coefficients (amplitude)

f(x,y) |F(𝟂)|
What does it mean to be at pixel x,y?
What does it mean to be more or less bright in the Fourier decomposition image?
Slide by Steve Seitz

87

43
 2/20/24

Fourier Bases

Teases away ‘fast vs. slow’ changes in the image.

Green = sine
Blue = cosine

This change of basis is the Fourier Transform

88

88

Fourier Bases

89

89

44
 2/20/24

2D Fourier Transform

Slide by Steve Seitz

90

Basis reconstruction

Danny Alexander
91

91

45
 2/20/24

Basis reconstruction

92

92

2D Fourier transform

Image Fourier Spectrum

Percentage of image power enclosed in circles (small to large) :

90, 95, 98, 99, 99.5, 99.9

Most of energy concentrated in low frequencies

93

93

46
 2/20/24

Image filtering in the frequential domain

• We will be dealing only with functions (images) of finite

duration so we will be interested only in Fourier Transform

94

94

Image filtering in frequential domain

FFT

FFT
=

Inverse FFT

Slide by Derek Hoiem

95

47
 2/20/24

Now we can edit frequencies!

96

96

Low-pass and high-pass filtering

Slide by Alexei Efros

97

48
 2/20/24

Low-pass filter

f(x,y) F(u,v)

spatial domain frequency domain

f(x,y) F(u,v)

G(u,v) = F(u,v) • H(u,v) •

filter
g(x,y) G(u,v)

Purpose: smoothing noise

g(x,y) H(u,v)

98

98

H(u,v) - Ideal low-pass filter

1 D(u,v) £ D0 D(u,v) = Ö u2 + v2
H(u,v) =
0 D(u,v) > D0 D0 = cut off frequency

H(u,v)
H(u,v)

1
v

u 0 D0 D(u,v)

99

99

49
 2/20/24

Blurring - Ideal low-pass filters

99.7% 99.37% 98.65%

100

100

The ringing problem

G(u,v) = F(u,v) • H(u,v)

Convolution Theorem

g(x,y) = f(x,y) * h(x,y)

IFFT sinc(x)

H(u,v) h(x,y)

­ D0 ¯ Ringing radius + ¯ blur

101

101

50
 2/20/24

The ringing problem

Freq. domain Spatial domain

250

200

150

100

50

0
0 50 100 150 200 250

102

102

H(u,v) - Gaussian filter

H(u,v) H(u,v)
1

1/ e

v
u 0 D0 D(u,v)

-D2(u,v)/(2D20)
H(u,v) = e D(u,v) = Ö u2 + v2

Softer Blurring + no Ringing

103

103

51
 2/20/24

Blurring - Gaussain lowpass filter

99.11% 98.74% 96.44%

104

104

The Gaussian lowpass filter

Freq. domain Spatial domain

300

250

200

150

100

50

0
0 50 100 150 200 250 300

105

105

52
 2/20/24

Blurring in the Spatial Domain

Averaging = convolution with 1 1 1 = point multiplication of the

111
transform with sinc:
1 2 1
Gaussian Averaging = convolution with 2 4 2
1 2 1
= point multiplication of the transform with a gaussian.
0.15

1
0.1 0.8

0.6
0.05 0.4

0.2
0 0
0 50 100 -50 0 50
Image Domain Frequency Domain

106

106

High-pass filter

H(u,v) - Ideal Filter

0 D(u,v) £ D0 D(u,v) = Ö u2 + v2
H(u,v) =
1 D(u,v) > D0 D0 = cut off frequency
H(u,v)
H(u,v)

v 1

u 0 D0 D(u,v)

107

107

53
 2/20/24

High-pass gaussian filter

H(u,v)
H(u,v)
1

v
1- 1 / e

u 0 D0 D(u,v)

-D2(u,v)/(2D20)
H(u,v) = 1 - e

D(u,v) = Ö u2 + v2

108

108

High-pass filtering

−1 −1 −1
ℎ&(3𝑥3) = −1 8 −1
−1 −1 −1

109

109

54
 2/20/24

High pass filtering

Original image Spectre with filters

110

110

High pass filtering

111

111

55
 2/20/24

Band-pass filtering

0 D(u,v) £ D0 - w D(u,v) = Ö u2 + v2
2
w w
H(u,v) = 1 D0- 2
£ D(u,v) £ D0 + 2
D0 = cut off frequency
w
0 D(u,v) > D0 + 2 w = band-width

H(u,v) H(u,v)

1
v
D(u,v)
0 D0- w D0 D0+w
2 2
u
Can be obtained by multiplying the filter functions of a low-pass and
of a high-pass in the frequency domain

112

112

Removing sinus noise

Brayer

113

56
 2/20/24

Removing sinus noise

114

High-pass filtering + orientation

115

57
 2/20/24

Hybrid Images

A. Oliva, A. Torralba, P.G. Schyns, SIGGRAPH 2006

116

Contents

• Remind: digital image

representation
• Point Operators
• Convolution and spatial
filtering
• More neighborhood operators
• Image transforms
• Frequency domain
• PCA

CV Group – School of Information and Communication and Technology

117
117

58
 2/20/24

Principle Component Analysis – PCA (KL transformation)

• PCA transforms the original input space into a lower

dimensional space
• By constructing dimensions that are linear combinations
of the given features

• The objective: consider independent dimensions

along which data have largest variance (i.e., greatest
variability)

Slide by Jana Kosecka

118

Principal Component Analysis (cont.)

• PCA enables transform a number of possibly correlated

variables into a smaller number of uncorrelated variables
called principal components

• The first principal component accounts for as much of the

variability in the data as possible

• Each succeeding component (orthogonal to the previous

ones) accounts for as much of the remaining variability as
possible

Slide by Jana Kosecka

119

59
 2/20/24

Principal Component Analysis (cont.)

• PCA is the most commonly used dimension reduction

technique.
• Data samples

• Compute the mean

• Computer the covariance matrix:

Slide by Jana Kosecka

120

Principal Component Analysis (cont.)

Slide by Jana Kosecka

121

60
 2/20/24

Principal Component Analysis (cont.)

Slide by Jana Kosecka

122

Principal Component Analysis (cont.)

Slide by Jana Kosecka

123

61
 2/20/24

Illustration of PCA

Slide by Jana Kosecka

124

Illustration of PCA

Slide by Jana Kosecka

125

62
 2/20/24

Determining the number of components

• Plot the eigenvalues

• each eigenvalue is related to the amount of variation
explained by the corresponding axis (eigenvector)

• If the points on the graph tend to level out (show an

“elbow” shape), these eigenvalues are usually close
enough to zero that they can be ignored

Slide by Jana Kosecka

126

The space of all face images

• When viewed as vectors of pixel values, face images are

extremely high-dimensional
• 100x100 image = 10,000 dimensions
• However, relatively few 10,000-dimensional vectors
correspond to valid face images
• We want to effectively model the subspace of face images

Slide by Jana Kosecka

127

63
 2/20/24

Eigenfaces: Key idea

• Assume that most face images lie on

a low-dimensional subspace determined by the first
k (k<d) directions of maximum variance
• Use PCA to determine the vectors or “eigenfaces”
u1,...,uk that span that subspace
• Represent all face images in the dataset as linear
combinations of eigenfaces

Slide by Jana Kosecka

128

Eigenfaces example- Traning images

Slide by Jana Kosecka

129

64
 2/20/24

Eigenfaces example

Slide by Jana Kosecka

130

Eigenfaces example

Slide by Jana Kosecka

131

65
 2/20/24

Eigenfaces examples

Slide by Jana Kosecka

132

Recognition with eigenfaces

Slide by Jana Kosecka

133

66
 2/20/24

THANK YOU !

CV Group – School of Information and Communication and Technology

134
134

67