Feature Detection

Template matching, coorrelation, features, corners

Feature Detection
We want to establish the concept of feature in an image.
We will start with a basic mechanism of matching patches from one
image to other image.
will define Sum of Square Distance
And show how it leads to correlation and cross-correlation
then construct case for normalized correlation
Look into HOW PATCH with corner IS MORE IMPORTANT THAN OTHERS

Using concept of auto-correlation

we will develop idea of how to detect corners

Correspondence Problem
Given two images you want to find what is similar?
for stereo
Matching

Correspondence across views

Correspondence: matching points, patches, edges, or regions across
images

Example: estimating fundamental matrix that

corresponds two views

Slide from Silvio Savarese

Example: structure from motion

Correspondence Problem
Given two images you want to find what is similar?
Following is SIFT key point detector with L2 matching
(can you see some errors?)

An old images of Lahore (possibly in Govt. College Lahore)

Source: Lahore Nama

Correspondence Problem
Given two images you want to find what is similar?
Following is SIFT key point detector with L2 matching
What is SIFT?

Patch Matching

SSD for template matching

8

10

10

10

10

SSD for template matching

8

10

10

10

10

2-8

8-9

36

8-9

2-3

1-2

5-9

16

Subtract each element

with their respective
element

Square each
element

56

SSD for template matching

8

10

10

10

10

210

Subtract each element

with their respective
element

Square each
element

SSD for template matching

8

10

10

10

10

Subtract each element

with their respective
element

Square each
element

SSD for template matching

8

10

10

10

10

Performing SSD on every

possible location

56

150

97

155

43

34

-1

184

29

145

24

163

140

-1

23

204

12

210

87

77

-1

203

164

62

107

52

-1

19

182

50

54

99

111

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

SSD for template matching

8

10

10

10

10

Performing SSD on every possible

location (below is Matlab Code)

D = -1*ones(size(A));
for i=1:5
for j = 1:6
D(i,j)= (sum(sum((B - A(i:i+2, j:j+1)).^2)));
end
end

56

150

97

155

43

34

-1

184

29

145

24

163

140

-1

23

204

12

210

87

77

-1

203

164

62

107

52

-1

19

182

50

54

99

111

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

-1

Relationship between SSD and Correlation

Type equation here.

Maximize this

g
g

c= 1 1 + 2 2 + + = .

Cross Correlation
When exhaustively search over all the patches of other image

Feature Detection

Feature Detection
We talked about how to compare one part of image
into other
Now we shall talk about
How to detect what is important in an image.
Locations or points which are important are called
interest points = keypoints

Many applications

tracking: which points are good to track?

recognition: find patches likely to tell us something about
object category
3D reconstruction: find correspondences across different
views

original

Suppose you have to

click on some point,
go away and come back
after I deform the
image,
and click on the same
points again.

Which points would you

choose?
deformed

Overview of Keypoint Matching

A1

A2

1. Find a set of
distinctive keypoints
2. Define a region
around each
keypoint

A3

fA

fB

d( f A, fB ) T

3. Extract and
normalize the
region content
4. Compute a local
descriptor from the
normalized region

5. Match local
descriptors
K. Grauman, B. Leibe

Goals for Keypoints

Detect points that are repeatable and distinctive

Key trade-offs

A1
A2

A3

Detection of interest points

More Repeatable
Robust detection
Precise localization

More Points
Robust to occlusion
Works with less texture

Description of patches
More Distinctive

More Flexible

Minimize wrong matches

Robust to expected variations

Maximize correct matches

What Are Image Features?

Global Features
A global property of an image, e.g. the average gray level

Local Features
A part of an image with some special property, e.g. circle, line, specific
texture etc.

Local Features - Definition

(Good) Image features are local, meaningful, detectable
parts of an image and should be invariant to normally
expected changes in the scene
Meaningful
Features associated to interesting scene elements via the image
formation process, e.g. intensity variations created by contours of an
object in the scene
# of pixels with pixel-level 134 is a feature which is not meaningful
(why?) and also not invariant to light changes

Detectable
Means that algorithms should exist to detect the location of that
feature, otherwise it is of no use

Feature Extraction
Commonly used features
Points
Edges
Contours

Feature Extraction is the process of finding these

features
Feature Extraction is often an intermediate step,
not the goal in itself

For example, point feature extractors are often used in

stereo to find the relative geometry of the cameras, and
then subsequently find depth

Invariant Local Features

Image content is transformed into local feature coordinates that are
invariant to translation, rotation, scale, and other imaging parameters

Features Descriptors

Back to Patch Matching

Basic Idea

flat region:
no change in
all directions

edge:
no change along
the edge direction

corner:
significant change
in all directions

www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/InvariantFeatures.ppt

Corner Detection: Basic Idea

We should easily recognize the point by looking through a small
window
Shifting a window in any direction should give a large change in
intensity

flat region:
no change in
all directions
Source: A. Efros

edge:
no change
along the edge
direction

corner:
significant
change in all
directions

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

Demo of a point + with well distinguished neighbourhood.

>0

I x2 I x I y
The matrix
is the covariance of the gradient
2
in a window I x I y I y
Because we can assume gradients to be zero mean

vectors

This matrix can be used to distinguish between corners, edges and plain
patches.
Recall: For a set of 2D vectors
covariance
We can write C as

s.t.

which is why the matrix at the top of the slide is covariance of gradient vector

77

78

79

80

81

84

85

The problem of Scale

86

The Problem of Scale

87

Solution to Scale Problem

Many scale invariant corner detectors have been proposed
They essentially find corners are multiple scales
A good corner detector, invariant to intensity and geometric
distortions is part of ongoing research

88

Feature extraction: Corners

9300 Harris Corners Pkwy, Charlotte, NC

Slides from Rick Szeliski, Svetlana Lazebnik, and Kristin Grauman

Detecting Point Features (Corners)

http://en.wikipedia.org/wiki/Corner_detection

corners detected

92