Feature-Based Image Alignment

Feature-Based Image Alignment

CS 650: Computer Vision
Bryan S. Morse

Feature-Based Image Alignment

Introduction

Image Registration

One common task is to align two images

I

Different positions

Different times

Different types of imaging

This is called image registration.

Feature-Based Image Alignment

Introduction

Image Stitching

One common application of image registration is to stitch together or

mosaic two or more images together.
Heres a link to an example.

Feature-Based Image Alignment

Introduction

Image Stitching
The key to image stitching is to be able to warp one image to match
another, then combine.
Cases:
I

Cant do in general without depth information due to parallax

Can do if all of the images have the same focal point

(pure 3-D rotation, panoramas)

Can do if scene is planar (homographies)

Feature-Based Image Alignment

Homographies

Planar Case
If the scene is planar or approximately so (a wall, objects far away,
etc.), you can warp one image to another using a homography:
x
0 = H

x
or

x0
y0
1

x 0
h00
0
y = h10
0
w
h20

h01
h11
h21

h02
x
h12 y
1
1

6= 0, the homography uniquely maps 2-D homogeneous

If det(H)
points to other 2-D homogeneous points.

Feature-Based Image Alignment

Homographies

Finding the Homography Between Two Images?

Common approaches to registration:

I

Area matching (direct or area-based)

Sparse set of feature points (feature-based)

Segment, represent, then match objects

can be very robust for multimodal registration but very
domain-specific

Feature-Based Image Alignment

Direct or Area-Based Approaches

Direct Registration
Basic approach:
I

Define some error metric that measures how well the images
match (correlation, L-norms, normalized cross-correlation, etc.)

Select the parameters for the homography that optimizes the

quality of the match

Well come back later to ways to do this.

Feature-Based Image Alignment

Feature-Based

Feature-Based Registration
Basic approach:
I

Find interesting points (Moravec, Harris, SIFT, etc.)

Match them somehow (neighborhoods, SIFT descriptors, etc.)

Solve for the homography

Can solve for a homography with a minumum of four points, with no

three of them colinear (the 4-point algorithm).
Warning: dont even get close to colinear!!
What about noise? What about mismatches?

Feature-Based Image Alignment

The Four-Point Algorithm

The Four-Point Algorithm

Mapping:

x0
h00
y 0 h10
1
h20

or
x0 =

h00 x + h01 y + h02

h20 x + h21 y + 1

h01
h11
h21

h02
x
h12 y
1
1
y0 =

h10 x + h11 y + h12

h20 x + h21 y + 1

y0 =

h10 x + h11 y + h12

h20 x + h21 y + 1

Feature-Based Image Alignment

The Four-Point Algorithm

The Four-Point Algorithm

Using
x0 =

h00 x + h01 y + h02

h20 x + h21 y + 1

Lets re-write this:

h00 x + h01 y + h02 x 0 (h20 x + h21 y + 1)

h10 x + h11 y + h12 y 0 (h20 x + h21 y + 1)

Can we write this in matrix form?

= 0
= 0

Feature-Based Image Alignment

The Four-Point Algorithm

The Four-Point Algorithm

x
0

y
0

1 0
0 x

0
y

0
1

x 0 x
y 0 x


0
x y

y 0 y

h00
h01
h02
h10
h11
h12
h20
h21

 0 

= x0

Feature-Based Image Alignment

The Four-Point Algorithm

The Four-Point Algorithm

x1
0
x2
0
x3
0
x4
0

y1
0
y2
0
y3
0
y4
0

1
0
1
0
1
0
1
0

0
x1
0
x2
0
x3
0
x4

0
y1
0
y2
0
y3
0
y4

0
1
0
1
0
1
0
1

x10 x1
y10 x1
x20 x2
y20 x2
x30 x3
y30 x3
x40 x4
y40 x4

x10 y1
y10 y1
x20 y2
y20 y2
x30 y3
y30 y3
x40 y4
y40 y4

h00
h01
h02
h10
h11
h12
h20
h21

x10
y10
x20
y20
x30
y30
x40
y40

Feature-Based Image Alignment

The Four-Point Algorithm

Camera Calibration (Revisited)

I

This general approach found in the four-point algorithm can be

used in lots of other problems.

Keep in mind that its the elements of the matrix that are the
unknowns and re-write the equations to make these the
unknown vector and the other information the matrix

Example: Camera Calibration

p P pw
Write in terms of 11 unknowns of P and then use known 3-D
points pw and projected points p

Feature-Based Image Alignment

Least-Squares Solvers

Dealing with Noise

Deal with noise in estimating any one point by using more points than
neededthis is an overconstrained system of equations
Let x0 = (x 0 , y 0 ) denote the remapped points as before:
x0 =

h00 x + h01 y + h02

h20 x + h21 y + 1

y0 =

h10 x + h11 y + h12

h20 x + h21 y + 1

and let x denote the correct matching position in the other image,
then minimize the sum of the squared error:
X
2
kx x k
i

and solve using least-squares fitting.

Feature-Based Image Alignment

Least-Squares Solvers

Least Squares
Your book has a more detailed discussion in 6.1.1, 6.1.3, A.2.
General Form:
Let for each point pair i and a set of transformation parameters p:
xi0 = f (xi , p)
xi
ri = xi xi0

predicted (mapped) location

measured location of matching point
error or residual

Goal: minimize the squared residuals

X
X
2
2
kri k =
kx 0 x k
i

Key piece: if xi stays fixed and you change a parameter in p, how

does the predicted matching point xi0 change?
Feature-Based Image Alignment
Least-Squares Solvers

Linear Least Squares

The Jacobian is an extension of the gradient for multiple outputs:
J=

f
p

If the relationship between the point motion and the parameters p is linear
(e.g., up to affine),
x = x 0 x = J(x) p

The minimum-error result can then be found by a pseudoinverse technique:

Ap=b

where
A

J T (xi )J(xi )

X
i

J T (xi )xi

Feature-Based Image Alignment

Least-Squares Solvers

Nonlinear Least Squares

If the relationship between the point motion and the parameters of the
transformation is not linear (e.g., homography, lots of other things in
vision), you have to use an iterative solver.
I

Basic idea: iteratively adjust parameters until residual reaches

minimum essentially variants on gradient descent

The Jacobian is essential to all of these since it tells you how

changing each parameter in p will change the result

One popular technique for minimizing squared-error measures is

Levenberg-Marquardt.

Iterative solvers require a good starting point, so how do you get

one? Use a linear technique to seed it (e.g., four-point method).

Feature-Based Image Alignment

Outliers

Dealing with Bad Point Matches

Some point matches might be wronghow do you deal with it?

From a least-squares fitting view, these are outliers, which throw

off the answer.

Common approach: Use RANSAC and the four-point algorithm

to simultaneously separate outliers and solve for initial estimate

for H.

Throw all the resulting inliers into a least-squares solver.