Lecture 10 (4.9.

07)

Image Description

Shahram Ebadollahi

4/11/2007 DIP ELEN E4830 1

Lecture Outline

1 Motion (cont.)
1 Texture & Texture Quantification
1 Moments
1 Boundary Description
1 Color Description
1 MPEG-7 Descriptors

4/11/2007 2
 Motion – object

Difference image:
d (i, j ) = 0 if | f1 (i, j ) − f 2 (i, j ) | ≤ ε object
motion
1 otherwise

No motion direction information !

f1 f2

n
d cum (i, j ) = ∑ ak | f1 (i, j ) − f k (i, j ) |
k =1 Cumulative difference image

Tells us how often the image gray

level was different from gray-level
of reference image

absolute positive negative

4/11/2007 3
Motion Field
• A velocity vector is assigned to each pixel in the image
• Velocities due to relative motion between camera and the 3D
scene
• Image change due to motion during a time interval dt
• Velocity field that represents 3-dimensional motion of object
points across 2-dimensional image

Motion field
4/11/2007 4
 Optical flow
• Motion of brightness patterns in image sequence

• Assumptions for computing optical flow:

• Observed brightness of any object point is
constant over time
• Nearby points in the image plane move in a
similar manner

∂f ∂f ∂f
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) + dx + dy + dt + O(∂ 2 )
∂x ∂y ∂t
Gray-level difference at same
= f ( x, y, t ) + f x dx + f y dy + f t dt + O(∂ )
2 location over time is
equivalent to product of spatial
gray-level difference and
dx dy velocity
f ( x + dx, y + dy, t + dt ) = f ( x, y, t ) ⇒ − f t = f x + fy
dt dt
− f t = f x u + f y v = ∇f .c
dx dy
c=( , ) = (u, v)
dt dt
4/11/2007 known unknown 5
Optical Flow Constraints

− f t = f xu + f y v

• no spatial change in brightness, induce no temporal

change in brightness 2 no discernible motion

• motion perpendicular to local gradient induce no temporal

change in brightness 2 no discernible motion

• motion in direction of local gradient, induce temporal

change in brightness 2 discernible motion

• only motion in direction of local gradient induces temporal

change in brightness and discernible motion

Aperture problem

4/11/2007 6
Optical flow != Motion Field

MF ≠ 0 MF = 0
OF = 0 OF ≠ 0

4/11/2007 7
MPEG-7 Motion Descriptors

1 Motion activity: slow-paced,

fast-paced, action
1 Features:
1 Intensity of activity
1 Direction of activity
1 Spatial distribution of
activity
1 Temporal distribution of
activity
4/11/2007 8
Texture - Definition

4/11/2007 9
 Texture – Quantification Methods

1 Statistical: compute local features at

each point in image and derive a set
of statistics from the distribution of
local features
1 1st, 2nd, and higher-order statistics
based on how many points are
used to define local features

1 Structural: texture is considered to

be composed of “texture elements”.
Properties of the “texture elements”
and their spatial placement rules
characterizes the texture
1 Original texture can be
reconstructed from its structural
description

4/11/2007 10
Statistical Texture Analysis
1st order statistics

image f → hf histogram

• Obtain statistics of the histogram:

n
Mean: ∑ ih(i)
i =1
n
Standard
deviation: ∑ (i − µ )
i =1
2
h(i )
n
Skewness: ∑ (i
i =1
− µ ) 3
h(i )
n
Entropy: − ∑ h(i ) log h(i ) Measure of uniformity of histogram
4/11/2007 i =1 11
 Statistical Texture Analysis
1st order statistics
image
histogram
statistics

Skewness = 2.08 Skewness = 2.44 Skewness = -0.092

Entropy = 0.88 Entropy = 0.77 Entropy = 0.97

4/11/2007 12
 Statistical Texture Analysis
2nd order statistics: Co-occurrence

f (m2 , n2 ) = j
d
θ P( d ,θ ) (i, j ) ≈ Pr[ f (m1 , n1 ) = i, f (m2 , n2 ) = j ]
f (m1 , n1 ) = i

1 2 j L
1 Joint gray-level histogram of pairs of 1
pixels 2
P( d ,θ ) ( i , j )
1 2D histogram

L
4/11/2007 13
 Statistical Texture Analysis
2nd order statistics: Co-occurrence
P( d ,θ =0° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
k − m = 0, | l − n | = d , f (k , l ) = i, f (m, n) = j} |
P( d ,θ = 45° ) (i, j ) =| {((k , l ), (m, n)) ∈ ( M × N ) × ( M × N ) :
(k − m = d , l − n = −d ) ∨ (k = m = −d , l − n = d ), f (k , l ) = i, f (m, n) = j} |

P( d ,θ =90° ) (i, j )
90°
P( d ,θ =135° ) (i, j )
135° 45°
k

{1} is set cardinality

0°
j
4/11/2007 14
Statistical Texture Analysis
2nd order statistics: Co-occurrence (example)
image Co-occurrence matrix

4/11/2007 15
 Statistical Texture Analysis
2nd order statistics: Co-occurrence (statistics)
L L

Angular 2nd moment (energy): ∑∑ ( d ,θ ) (i, j )

P 2

i =1 j =1
(measure of image homogeneity)

Maximum Probability: max P( d ,θ ) (i, j )

i, j

L L
− ∑∑ P( d ,θ ) (i, j ) logP( d ,θ ) (i, j )
Entropy: i =1 j =1

L L

∑∑ λ κ
i − j P( d ,θ ) (i , j )
Contrast: i =1 j =1
(measure of local variations) L L

∑∑ [ijP
i =1 j =1
( d ,θ ) (i, j )] − µ x µ y
Correlation:
(measure of image linearity) σ xσ y
L L L L
4/11/2007 µ x = ∑ i ∑ P( d ,θ ) (i, j ) σ x = ∑ (i − µ x ) 2
∑P ( d ,θ ) (i, j ) 16
i =1 j =1 i =1 j =1
 Statistical Texture Analysis
2nd order statistics: Difference Statistics
P( d ,θ ) (k ) = ∑P ( d ,θ )
i , j∈{1,1, L}
(i, j ) is a subset of co-occurrence matrix

|i − j | = k

L −1

Angular 2nd moment (energy): ∑ (d ,θ ) (k )

P 2

k =0

L −1
Mean: ∑ kP
k =0
( d ,θ ) (k )

L −1

Entropy: − ∑ P( d ,θ ) (k ) log P( d ,θ ) (k )
k =0

L −1
Contrast: ∑ P(d ,θ ) (k )
k 2

k =0

4/11/2007 17
Statistical Texture Analysis
2nd order statistics: Fourier Power Spectrum
f ( x, y ) ↔ F (u , v)

Power Spectrum P(u, v) =| F (u , v) |2

v
π Indicator for size of
P ( r ) = 2∑ P ( r , θ ) dominant texture element
u or texture coarseness
θ =0

L/2
v
P(θ ) = ∑ P(r , θ )
Indicator for the
r =0
directionality of the texture
u
4/11/2007 18
 Statistical Texture Analysis
2nd order statistics: Autocorrelation

M − pN −q

MN ∑ ∑ f ( k , l ) f ( k + p, l + q )
C ff ( p, q ) = k =1 l =1
( M − p )( N − q ) M N

∑∑
k =1 l =1
f 2 (k , l )

Large texture elements 3 autoccorrelation decreases slowly with increasing distance

Small texture elements 3 autoccorrelation decreases rapidly with increasing distance

Periodic texture elements 3 periodic increase & decrease in autocorrelation value

4/11/2007 19
MPEG-7 Texture Descriptors

image
1 Homogeneous texture
descriptors
1 Response of bank of
filters sensitive to image
scale and orientation
1 Non-homogeneous
texture descriptors • vertical
• horizontal
(edge histogram) • 45 degrees
1 Spatial distribution of • 135 degrees
edges • non-directional edge

4/11/2007 20
Structural Texture

4/11/2007 21
Structural Texture

4/11/2007 22
Geometric Moments

m pq = ∫ ∫ x p y q f ( x, y )dxdy (p+q)-th 2D geometric moment

Projection of f(x,y) onto monomial x p yq

• Why use moments?

• Geometric moments of different orders represent spatial
characteristics of the image intensity distribution

m00 Total intensity of image. For binary image 2 area

x0 = m10 / m00 Intensity centroid

y0 = m01 / m 00 binary image 2 geometrical center

4/11/2007 23
 Central Moments
Elliptical shape descriptors
µ pq = ∫ ∫ ( x − x0 ) ( y − y0 ) f ( x, y )dxdy
p q

Image ellipse characterizes

fundamental shape features and
µ 00 = m00 µ10 = µ 01 = 0 also 2D position and orientation

µ 02 , µ 20 Variance about the centroid

µ11 covariance
θ
Principal moment of inertia: 2µ11
θ = 0.5 tan −1 ( )
( µ 20 + µ 02 ) + [( µ 20 − µ 02 ) 2 + 4µ112 ]1/ 2 µ 20 − µ 02
I1 =
2
a = 2( I1 / µ 00 )1/ 2 b = 2( I 2 / µ 00 )1/ 2
( µ 20 + µ 02 ) − [( µ 20 − µ 02 ) 2 + 4µ112 ]1/ 2
I2 =
2
( I1 + I 2 ) / m00
2
spreadness

4/11/2007
( I 2 − I1 ) /( I1 + I 2 ) elongation 24
Geometric Moment Invariants
 x'  a11 a12   x 
( x, y ) ( x' , y' )  y ' = a a22   y 
   21

m' pq = ∫ ∫ (a11 x + a12 y ) p (a21 x + a22 y ) q f ( x, y )∆dxdy ∆ = a11a22 − a21a12

Scale

x' = kx y ' = ky m' pq = k p + q + 2 m pq

η pq = µ pq /( µ00 ) ( p + q + 2 ) / 2 Scale and translation invariant

4/11/2007 25
Boundary Representation –
Chain Codes

4/11/2007 26
Color descriptors
Color histogram

• Dominant color descriptor

• cluster colors
• descriptor = representative colors, their percentage, color variance

4/11/2007 27
Which descriptors?
Image Feature Evaluation

1. Prototype Performance
1 Classify (Segment) image using
different features
1 Evaluate which feature is optimal
(minimum classification error)

2. Figure of Merit
1 Establish functional distance
measurements between set of
image features (large distance 2
low classification error)
1 Bhattacharyya distance

4/11/2007 28