Chapter 5: Image processing

Edge
dge detection
de ec o

I
Image segmentation
i

Corner detection
Introduction

Image analysis ---

--->> extracting information from an image

Feature extraction
segmentation

Subdivides image into its constituent parts or objects

Content
Introduction
Detection of discontinuities (edge detction)
Masks
M k
Canny
Hough transform
Cartesien coordinate
Polar coordinate
Similarity
Histogram based method
Split and merge
Region growing
Corner detection
Two p
points of view

Based on basic properties of gray level values

Discontinuity
y Similarity
y

partition partition

Based on abrupt changes in Thresholding,

gray levels region growing,
li
lines - edges
d region
i splitting
litti and
d merging
i
Example
p

similarities

discontinuities

Satellite image of marseille

1. Detection of discontinuities

Three basic types of discontinuities : points, lines, edges

Common way : run a mask through the image.

Sum of products of the coefficients with the gray filtering
l
levels
l contained
t i d iin th
the region
i encompassed dbby the
th
mask

R = ∑i =1Wi I i
W1 W2 W3
9
W4 W5 W6

W7 W8 W9
Gray level

Mask coefficient
1.1. Point detection
Measure of the weighted difference
-1 -1 -1 between the center point and its
neighbors
g
-1 8 -1

-1 -1 -1
HP filter
filt + threshold
th h ld

Point mask

R >T T = non negative threshold value

Mask is a Modelisation of a point

1.2. Line detection
-1 -1 -1 -1 -1 2 -1 2 -1 2 -1 -1

2 2 2 -1 2 -1 -1 2 -1 -1 2 -1

-1
1 -1
1 -1
1 2 -1
1 -1
1 -1
1 2 -1
1 -1
1 -1
1 2

H +45° V -45°
Line masks

If the H mask is moved around an image it would respond more

strongly to lines oriented horizontaly
Ex : try it with a background ‘ 1 ’ and a line with value ‘ 10 ’

i≠j that point is said to be more

For a pixel position, if |Ri|>|Rj| for all i≠
likely associated with a line in direction of mask i

M
More generall : template
t l t matching
t hi
correlation between signal and filters (masks, forms)
1.3. Edge
g detection

Detection of discontinuities in images, in gray levels

EDGE = boundary between 2 regions with relatively distinct gray levels.

(non homogeneous)

Model :

e
h lenght
e ϑ h width

Edge = high frequencies Realistic case

Edges:
Edge points are pixels at or around which the image values undergo a sharp variation –
pixels with large gradient.
There are 3 types
yp of edges:
g
Step edges -Ridges edges

Roof edges
g
Effects of noise
Consider a single row or column of the
image
Plotting intensity as a function of position
gives a signal

Where is the edge?

Solution: smooth first

Where is the edge?

Derivative theorem of convolution
This saves us one operation:
Laplacian
p of Gaussian
Consider

Laplacian of Gaussian
operator

Where is the Z
Zero-crossings
i off
edge?
Example
p
Image
50

20
40
First
derivative
30

40 20

10
60
0

80 -10

-20
Sign
100
-30
position
120 -40

20 40 60 80 100 120 -50

0 20 40 60 80 100 120 140

Profile of horizontal line

25

20
Second
15 derivative
200

10

180 5

0
160

-5

140
Position
P iti
-10
zero crossing
-15
120

-20

100
-25
0 20 40 60 80 100 120 140
0 20 40 60 80 100 120 140
Gradient
⎡∂∂ff ⎤
At position (x,y) of f(x,y) r ⎡Gx ⎤ ⎢ ∂x ⎥
∇f = ⎢ ⎥ = ∂f vector
⎣Gyy ⎦ ⎢ ∂y ⎥
⎢⎣ ⎥⎦
2 images

( )
r
magnitude ∇f = mag ∇f = Gx 2 + Gy 2 >0
or ∇f ≈ Gx + Gy for hardware 1 image

direction
α ( x, y ) = tan −1 (Gy Gx ) 1 image

Rem : Sobel provides both - differencing

- smoothing g effect !
Rem : derivatives may be implemented in digital form in several ways
Example
p
50
50

100
100

150
150

200
200 X
250
50 100 150 200 250
250
50 100 150 200 250

50
50

100
100
Y
150
150

200
200

250
50 100 150 200 250 250
50 100 150 200 250
Implementation
p
* approximation
i ti ∂f ≈1 f ( x + 1, y ) −1 f ( x, y ) -1
∂x H1
1
∂f ≈1 f (x, y + 1) − 1 f ( x, y )
∂y -1 1 H2

Other possibilities

* filtering f ⊗ H1 ⎫
⎬→ ( f ⊗ H1 )2 + ( f ⊗ H 2 )2
f ⊗ H2 ⎭

With noise attenuation; smooth in ⊥ direction of derivation

Examples
p of filters

0 -1 -1 0
2x2 filters
H1 H2 Roberts
1 0 0 1

3x3 filters -1
1 -c -1
1
-1 0 1

H1 -cc 0 c 0 0 0
H2

-1 0 1 1 c 1

C=1 Prewitt
C=1
C=2
C=2 Sobel

Mean = 0
Gradient template
p

Simple, general
noise sensitivity
threshold problem

Image of gradient = world of mountains and ridges

Threshold
T value ?

T1

|G(I)| T2

T1 -->
--> only one edge
T2 --
-->> two edges but edge 1 is large
See also zero crossing
Algorithm
g

1° convolve image by filters

2° calculate magnitude
3° threshold
th h ld
( ,y) =1 if G(x,y)≥t
⌧S(x,y) ( ,y)
Remarks:
R k
= 0 elsewere simple,
general
general,

4° edge linking
sensibility to noise
choice of t
thickness of egdes
2 filters!
Laplacian
p

At position (x,y) of f(x,y) ∂ 2

f ∂ 2
f
∇2 f = 2 + 2
∂ x ∂ y
1 image

In digital case ∇ 2 f = 4 z 5 − ( z 2 + z 4 + z 6 + z8 ) Zi: image intensities

1 filter !
0 -1
1 0 ΣCoefs=
C f 0 HP
Coefs=0
-1 4 -1

0 -1
1 -0
0
Laplacian
p

Find location (not direction) of edges using a

zero crossing property + sign

Properties
easy to implement
S ( x, y ) = ∇ 2 [G ( x, y ) ⊗ F ( x, y )]
edges : thickness = 1 pixel
robust
Gaussian filter Input image
detect every change of intensity

too much sensitive to noise

produces double edges

Improvement
Laplacian of 2D Gaussian function
Laplacian
p of ggaussian

S ( x, y ) = ∇ [G ( x, y ) ∗ f ( x, y )]
Using the properties
2

2nd derivative of smoothed version of f(x,y)

⎛ x2 + y2 ⎞
G ( x, y ) = exp⎜⎜ − ⎟⎟
⎝ σ 2
⎠ 1

σ=2
r −σ
2 2
⎛ r ⎞
2
∇ G=
2
p⎜⎜ − 2 ⎟⎟
exp
σ ⎝ 2σ ⎠
0.5
2

0
r

S ( x, y ) = ∇ G ∗ f ( x, y )
2
-0.5
-10 -8 -6 -4 -2 0 2 4 6 8 10
Image Segmentation
Image Segmentation
Comparison
p
Cannyy edge
g detector

Principle
Image features:

The image is characterized by :

- Global features :including intensity histogram,
histogram frequency domain descriptors,
descriptors
covariance matrix etc………
- Local features :including edges, corners, lines, curves, regions with special properties,
etc.
Depending on applications, various features are useful. We will focus on edges and
specially on Canny edge detector

Edge detection:

The aim of edge

g detection is to:
• Denoise
- Suppress image noise without loosing the real edges
• Edge
d enhancement
h
- Use filters that give only large responses to edges
• Edge localization
“Canny edge detector”:

1-Good Detection: it must minimize the probability of false positives as well as false
negatives . So it is the ability to locate and mark all real edges.

2-Good Localization: The edges detected must be as close as possible to the true edges.
So we have to obtain minimal distance between the detected edge and real edge.
edge

g Response
3-Single p or clear response:
p The detector must return one p
point onlyy for each
edge point, only one response per edge.

Given an image I:

St 1 Apply
Step A l CANNY_ENHANCER t I, output
CANNY ENHANCER to t t IE

NONMAX SUPPRESSION to IE , output IM

Step 2 Apply NONMAX_SUPPRESSION

Step 3 Apply HYSTERESIS_THRESH to IM, output IH

Algorithme de canny:
Step 1 : Filter
Convolution with Gaussian: (applying of Gaussian filter to denoise & enhance
edges)

The aim is to filter out any noise in the original image before trying to locate and detect any
edges. And because the Gaussian filter can be computed using a simple mask, it is used
exclusively in the Canny algorithm. It ‘s used to eliminate noise in reason to obtain a smooth
image.
The larger the width of the Gaussian mask, the lower is the detector
detector'ss sensitivity to noise.
The Gaussian mask used is:.

After the applying of the filter we’ll obtain:

J= G*I
Noise cancelling
noisy lena filtered with sigma = 3 filtered with sigma =1
 For pixel (i,j)
Step 2: Compute gradient component::
One estimating the gradient in the x-direction (columns) Gx
and the other estimating
g the gradient
g in the y
y-direction ((rows).
) Gy
y

I
Image G
Gx G
Gy
Estimate the edge strength:

The magnitude, or EDGE STRENGTH, of the gradient is then

approximated using the formula:

Es(i,j)=|G| = |Gx| + |Gy|

this formula is applied for each pixel
Step 3: Estimate the edge normal direction :

Finding the edge direction is trivial once the gradient in the x and y directions are known.

The
h fformula
l for
f finding
fi di the
h edge
d direction
di i is i just
j :
E0 (i,j)=theta = arctan (Gy / Gx)

The output is the strength image and an orientation image.

image
The result of CANNY_ENHANCER contain wide ridgesn around local maximums,
nonmaximum suppression will produce 1-pixel wide edges.
Find the direction matrix

Quantize E0 (i,j) into four directions, 0°, 45°, 90°, 135°

There are only four possible directions when describing the surrounding pixels 0degrees
(i th
(in the horizontal
h i t l direction),
di ti ) 45 degrees
d ( l
(along the
th positive
iti diagonal),
di l)
90 degrees (in the vertical direction), or 135 degrees (along the negative diagonal). So
now the edge orientation has to be resolved into one of these four directions depending
on which direction it is closest to.
Any edge direction falling within the yellow range (0 to 22.5 & 157.5 to 180
d
degrees) ) iis sett to
t 0 degrees.
d
Any edge direction falling in the green range (22.5 to 67.5 degrees) is set to 45
degrees.
degrees
Any edge direction falling in the blue range (67.5 to 112.5 degrees) is set to 90
degrees.
And finally, any edge direction falling within the red range (112.5 to 157.5 degrees)
is set to 135 degrees.

Example: showing CANNY_ENHANCER after nonmaximum suppression

Step
p 4 : Non maximum suppresion
pp
If the gradient is less than of the gradient of his neighbors,
neighbors
Using the direction of gradient, put the gradient in (x,y) à 0

Direction du gradient

(x,y)
Voisins
V i i lle llong
du gradient
Step 4 Hysteresis Thresholding:
Finally,
Fi ll hysteresis
h i is
i usedd as a means off eliminating
li i i streakingki & to remove
edge pixels caused by noise.
St ki is
Streaking i the
th breaking
b ki up off an edged contour t causedd by
b the
th operator
t
output fluctuating above and below the threshold. Some edge pixels has
l
lower values
l than
th false
f l edge
d pixels
i l so we have
h to
t eliminate
li i t them
th byb using
i
this method of thresholding.

Canny hysteresis thresholding - results

The Cannyy edge
g detector

thinning
(non-maximum suppression)
Non-maximum suppression
pp

Check if pixel is local maximum along

gradient direction
requires checking interpolated pixels p and r
 Predicting
the
h next
edge point
Assume the
marked point is
an edge
d point.
i
Then we
construct the
tangent to the
edge curve
(which is
normal to the
gradient at that
point) and use
this to predict
p
the next points
(here either r
or s).
)
(Forsyth & Ponce)
Hysteresis
y

Check that maximum value of gradient

value is sufficiently large
drop-outs? use hysteresis
⌧use a high threshold to start edge curves and a low
th h ld to
threshold t continue
ti them.
th
Effect of σ ((Gaussian kernel size))

original Canny with Canny with

The choice of depends on desired

behavior
large detects large scale edges
small detects fine features
Scale
Smoothing
Eliminates noise edges.
Makes edges smoother.
Removes fine detail.
(Forsyth & Ponce)
fine scale
high
threshold
coarse
scale,
high
threshold
coarse
scale
low
threshold
Edge
g linkingg

Ideally
previous technics yield pixel lying on the boundary between regions

In practice
set of pixels seldom characterizes a boundary completely
(because of noise)
breaks in the boundaryy

Local processing

Global processing
Hough
g transform
EL : local processing
p g
Si l approach
Simple h for
f linking
li ki edges
d is
i to
t
analyse characteristics of pixels in a small neighborhood 3x3, 5x5 for
all (x,y) that has undergone edge detection

all points that are similar are linked

1° strenght of response of gradient operator

2° direction of the gradient

(x,y) & (x ’,y ’) in the same neighborhood if

∇f ( x, y ) − ∇f ( x ' , y ') ≤ T T>0

T 0

angle
α ( x , y ) − α ( x ' , y ') ≤ A Angle threshold

Rem : direction of the edge at (x,y) is ⊥ to the direction of gradient vector

Example
p of ellp
p

Sobel
T>25
T>25,, a=
a=15
15°°

Goal : find the licence plate

Chain codes

Pixels --
-->> Codes Chains

Direction vectors between successive boundary pixels are encoded

3 2 1 A
1 7
0
4 1
0 6 6
2
4 0
2 5
5 6 7
3 5
4

1/ start at any boundary pixel A

2/ find the nearest edge
g ppixel and code its orientation. In case of tie,,
choose the one with largest (or smallest) code value.
3/ continue until there are no more boundary pixels
Example
p

y Theta code
7 111
6 110
0 000
t 1 001
ϑ
0 000
6 110
x 5 101
ϑ
5 101
4 100
3 011
2 010
4 100
2 010
1 001
t

Generalized chain coding + A(x,y)

 Transformée de Hough
g
• Given a set of collinear edge points, each of them have associated a line in
pparameter space.
p
– These lines intersect at the point (a, b) corresponding to the parameters of the
line in the image space.
Pl d
Plan des parametres
t Pl de
Plan d l’i
l’image
b = − xa + y y = ax + b
b
y

a x
Infinités des droites Une droite y=f(x)
qui passent par un point fixe Plusieurs points col
col.
Transformée de Hough
g

a
 Hough Transform Technique
• At each point of the (discrete) parameter space,
space
count how many lines pass through it.
– Use an array
arra of co
counters
nters
– Can be thought as a “ parameter image”
• The higher the count, the more edges are collinear
in the image
g space.
p
– Find a peak in the counter array
– This is a “bright”
bright point in the parameter image
– It can be found by thresholding
 Transformée de Hough
g
Algorithme case (u,v) => accumulateur
bmin
i
1)Quadrillage de l’espace (a,b)
amin
2) P
Pour chaque
h case A(
A(u,v)=0
) 0
b
3) P
Pour chaque
h point
i t im
image(edge)
g ( dg )

a)
) p
pour a ∈ [a a ]
min max
b) Calculer b = − xa + y

c) Calculer la case (u,v) amax

u=int(a/da) v=int(b/db)
u=int(a/da), bmax
d) A(u,v)= A(u,v)+1 a
4) Pour toute case (u,v), si A(u,v)>seuil, cette case
représente une droite
Transformée de Hough
g

Problème lié à l’implémentation précédente

y = ax + b b = − xa + y
y b

a=100
a=11
a=10 a=10

x a=11 a=100
a
x et y fixés
Quadrillage impossible
 Transformée de Hough
g

Solution : une autre représentation

p des droites

y = ax + b x cos(θ ) + y sin(θ ) = ρ
y ρmin
ρ
θ −π/2
π/2 π/2 θ

x ρmax
θ Is th
the line
li orientation
i t ti
ρ
ρ Is the distance between
Espace des paramètres
the origin and the line
 New Parameter Space
• Use the parameter space (ρ, θ)
• Thee new
ew space iss FINITE
N
– 0 < ρ < D , where D is the image diagonal.
–0<θ<π
• The new space can represent all lines
– Y = k is represented with ρ = k, θ=90
– X = k is represented with ρ = k, θ=0
 Transformée de Hough
g
A Point in Image Space is now
represented as a SINUSOID
x cos(θ ) + y sin(θ ) = ρ ρ = x cos θ+y sinθ
sinθ

y
ρ
θ
θ

ρ
Espace image E
Espace des
d paramètres
èt
 Divvying Up Space
r = x cosθ + y sinθ “normal” form

imsiz
y coord e

dist r

x coord
0 g θ
angle π
Edge Image Parameter space
 Issues
i actually
in ll getting
i the
h
curve...

imsiz
y coord e

dist r

x coord
0 g θ
angle 2π
Edge Image Parameter space
 Hough Transform Algorithm
Input is an edge image (E(i,j)=1
(E(i,j) 1 for edges)

1 Di
1. i θ and
Discretize d ρ in
i increments
i off dθ and
d dρ.
d Let
L
A(R,T) be an array of integer accumulators, initialized
to 0.
2 For
2. F each
h pixel
i l E(i,j)=1
E(i j) 1 and
d h=1,2,…T
h 1 2 T do
d
1. ρ = i cos(h * dθ ) + j sin(h * dθ )
2. Find closest integer k corresponding to ρ
3
3. Increment counter A(h,k)
A(h k) by one
3. Find local maxima in A(R,T)
Transformée de Hough
g
50

100

150

200

250

300

350
50 100 150 200 250 300
50

100

150

200

250 50

300 100

350 150
50 100 150 200 250 300

200

250

300

350
50 100 150 200 250 300
 Segmentation

• Region based segmentation

2. Region
g based segmentation
g
Objective
j :p
partition image
g into regions
g

Basic formulae
let R entire image region
Partition R into n subregions R1,R2,
R1 R2 …, Rn
such that:
⌧∪Ri=R Segmentation is complete

⌧Ri is a connected region

g
⌧Ri∩Rj=∅ for all i and j i≠j
Regions are disjoints
⌧P(Ri)=true for i=1,2,…,n
i=1 2 n and
⌧P(Ri ∪ Rj)=false for i≠j

LOGICAL PREDICATE ex:same intensity

1- Similarities
Thresholding
Region oriented (region splitting-merging, agregation)

Hyp : gray level histogram of an image composed of light objects on

a dark background (2 dominant modes)

Frequency
Frequency

Pixel values Pixel values

T1 T2
T

One obvious way to extract objects ⎧ f ( x, y ) < T object 1

⎨
⎩ f ( x, y ) > T
is to select a thresthold T
background 0
Optimal
p thresholdingg
S
Suppose th
thatt an image
i contains
t i 2 values
l + gaussian
i noise
i
A priori proba of the two levels

p ( z ) = P1 p1 ( z ) + P2 p2 ( z )
Global pdf Zone 2 pdf
Zone 1 pdf

P1 z − μ1 2 P2 z − μ2 2
p( z ) = exp(− ) + exp(− )
2π σ 1 2σ 1 2π σ 2 2σ 2
P1 + P2 = 1

Hyp : dark region = background = 1 μ1 < μ 2

bright region = objects = 2

T?
Now pb of error : problem of classifying object points as background point

T +∞
E1 (T ) = ∫ p2 ( z )dz and E2 (T ) = ∫ p1 ( z )dz
−∞ T

Overall prob error

E (T ) = P2 E1 (T ) + P1 E2 (T )

Find T value for which this error is minimal

Differentiating E(T) with respect to T and equating result to 0)

p ( z ) = P1 p1 ( z ) + P2 p2 ( z ) P1 p1 (T ) = P2 p2 (T )
 For gaussian density, taking logarithm,

AT 2 + BT + C = 0
A = σ −σ2
1
2
2

(
B = 2 μ1σ 22 − μ 2σ 12 )
C = μ 2 σ − μ σ + 2σ σ ln(σ 2 P1 / σ 1 P2 )
2 2 2 2 2 2
1 1 2 1 2

if
σ 12 = σ 22 = σ 2
μ1 + μ 2 σ2
T= + ln(( P2 / P1 )
2 μ1 − μ 2 A d if
And P1 = P2
μ1 + μ 2
T=
2
Example
p
2. Region
g splitting
p g and merging
g g

Subdivide an image initially into a set of

arbitrary disjointed regions
and then
MERGE and/or SPLIT the regions in an attempt
to satisfy previously defined constraints

Ex : quadtree
Q
Quadtree
Split into 4 disjoint quadrants any region
Ri where P(Ri)=False
Merge any adjacent regions Rj and Rk for which
P(Rj∪Rk)= TRUE
P(Rj∪Rk)
Stop when no further merging or splitting is
possible
ibl
R L f/b
Leaf/branch
h
R1 R2
R1 R2 R3 R4

R41 R42
R3 R41 R42 R43 R44
R43 R44
quadtree
partition
Example
p
4 steps

result
3- Region
g growing
g g pixel
p agregation
g g

Procedure that groups pixels or subregions

into larger regions
1) start with a set of seeds pts
2) grow region by appending to each seed
those neighboring pixels that have similar
properties
p p (gray
(g y level,, stats,, …))
Example
p
A A B B B
0 0 5 6 7
A A B B B
1 1 5 8 7
A A B B B
T=3
0 1 6 7 7
A A B B B
2 0 7 6 6
A A B B B

0 1 5 6 5

A A A A A
2 seeds
A A A A A

A A A A A
T=8
A A A A A
R1 associated to s1
R2 associated to s2 A A A A A

P
Property
t |zi-sj|<T
| i j|<T
Difficulties
⌧selection of initial seeds
military images with targets
⌧criteria choice
histogram, a priori info
pp g rule
⌧stopping
Image
g pprocessingg

Corner detection
 Corner detection

Corners contain more edges than lines.

A point on a line is hard to match.

Corners contain more edges
g than lines.

A corner is easier
Edge
g Detectors Tend to Fail at Corners
Findingg Corners

Intuition:
• Right at corner, gradient is ill defined.
• Near corner, gradient has two different
values.
values
•Ix a
and
d Iyy are
a e the
t e gradients
g ad e ts along
ao gXaand
dY
Formula for Findingg Corners

We look at matrix:
Gradient with respect to x, times
Sum over a small region, the gradient with respect to y
h
hypothetical
th ti l corner

⎡ ∑I 2
∑I I ⎤
C=⎢ x x y
⎥
⎢⎣ ∑ I x I y ∑I 2
y ⎥⎦

Matrix is symmetric WHY THIS?

 First consider case where:
First,

⎡ ∑I 2
∑I I ⎤ ⎡λ1 0⎤
C=⎢ x x y
⎥=⎢ ⎥
⎢⎣ ∑ I x I y ∑I λ2 ⎦
2
y ⎥⎦ ⎣ 0
Thi means allll gradients
This di t iin neighborhood
i hb h d are:
((k,0)
, ) or (0,
( , c)) or (0,
( , 0)) (or
( off-diagonals
g cancel).
)
What is region like if:
1. λ1 = 0? edge
2 λ2 = 0? edge
2.
3. λ1 = 0 and λ2 = 0? Homogeneous Region
4. λ1 > 0 and λ2 > 0? Corner
So,, to detect corners

Filter image.
Compute magnitude of the gradient
everywhere.
everywhere
We construct C in a window.
Use Linear Algebra to find λ1 and λ2.
If they are both big, we have a corner.