Image Registration

Mapping of Evolution

I2(x,y)=g(I1(f(x,y))
Registration Goals
f() 2D spatial transformation
g() 1D intensity transformation
(f(x,y))
2Dthe
spatial
transformation
f()
Assume
correspondences
are known
such
1D f()
intensity
transformation
g()
Find
and g() such
that the images
are best matched

Spatial Transformations

Rigid
Affine
Projective
Perspective
Global Polynomial
Spline

Rigid Transformation
Rotation(R)
Translation(t)
Similarity(scale)

x1
p1
y1

x
p2 2
y2

p2 t s Rp1

s1
s1
s2

cos( ) sin( )
R

sin(

)
cos(

t1
t1
t 2

Affine Transformation

Rotation
Translation
Scale
Shear

x2 a13 a11 a12 x1

y a a a y
2 23 21 22 1

No more preservation of
lengths and angles
Parallel lines are preserved

Perspective Transformation
(Planar Homography)

Perspective Transformation(2)
(xo,yo,zo) world coordinates
(xi,yi) image coordinates
fxo
xi
zo f

fyo
yi
zo f

Flat plane tilted with respect to the camera

requires Projective Transformation

Projective Transformation
(xp,yp) Plane Coordinates
(xi,yi) Image Coordinates
xi

a11 x p a12 y p a13

a31 x p a32 y p a33

yi

a21 x p a22 y p a23

a31 x p a32 y p a33

amn coefficients from the equations of the

scene and the image planes

Complex Transformations

Global Polynomial Transformation(splines)

Methods of Registration
Correlation
Fourier
Point Mapping

Correlation Based Techniques

Given a two images T & I, 2D normalized
correlation function measures the similarity for
each translation in an image patch
T ( x, y ) I ( x u , y v )

C (u , v )
I ( x u, y v)
x

Correlation must be normalized to avoid

contributions from local image intensities.

Correlation Theorem
Fourier transform of the correlation of two
images is the product of the Fourier
transform of one image and the complex
conjugate of the Fourier transform of the
other.

Fourier Transform Based Methods

Phase-Correlation
Cross power spectrum
Power cepstrum

All Fourier based methods are very efficient, only

only work in cases of rigid transformation

Point Mapping Registration

Control Points
Point Mapping with Feedback
Global Polynomial

Control Points
Intrinsic
Markers within
the Image

Extrinsic
Manually or Automatically
selected

After the control points have been

determined, cross correlation, convex hull
edges and other common methods are used
to register the sets of control points

Point mapping with Feedback

Clustering example: determine the optimal
spatial transformation between images by
an evaluation of all possible pairs of
feature matches.
Initialize a point in cluster space for each
transformation
Use the transformation that is closest to
the best cluster
Too many points, thus use a subset

Global Polynomial
Transformation(1)
Use a set of patched points to generate a
single optimal transformation
Bi-Variate transformation:

(x,y) reference image

u alj x y
l

j 1

l 0 j 0
m

v blj x y
l 0 j 0

j 1

(u,v) working image

Global Polynomial
Transformation(2)

When is polynomial transformation bad?

Splines approximate polynomial
transformations(B-spline, TP-spline)

Characteristics of Registration
Methods
Feature Space
Similarity Metrics
Search Strategy

Feature Spaces

Similarity Metrics

Search Strategies

Robust Multi-Sensor Image

Alignment
Irani & Anandan
Direct Method(vs. Feature Based)

Multi Sensor Images

EO

IR
Find features

Original Image
(Intensity Map)
Assume global
statistical
correlation

Loss of important
information
Often violated

Multi Sensor Image

Representation(1)
Same Modality Camera Sensors
enough correlated structure at all
resolution levels
Different Modality Camera Sensors
primary correlation only in high resolution
levels

Multi Sensor Image

Representation(2)

Goal: Suppress non-common information &

capture the common scene details
Solution: High pass energy images

Laplacian Energy Images

Apply the Laplacian high pass filter to the
original images
Square the results NO contrast reversal
BUT
The Laplacian is directionally invariant

Directional Derivative Energy

Images
Filter with Gaussian
Apply directional
derivative filter to the
original image in 4
directions
Square the resultant
images

Alignment Algorithm
Do not assume global correlation, use only
local correlation information
Use Normalized Correlation as a similarity
measure
Thus, no assumptions about the original
data

Behavior of Normalized Correlation

with Energy Images
NC=1
Two images are linearly related
NC<1(high)
Two images are not linearly related, yet
local fluctuations are low
NC<1(low)0
Incorrect displacements

Global Alignment with Local

Correlation(1)
a, b original images
{ai,bi} directional derivatives (i=1..4)
p=(p1p6)T affine
(u,v) shift from one image to another
Si(x,y)(u,v) correlation surface at a pixel(x,y)

S i( x , y ) (u , v) ai ( x, y ) Nbi ( x u , y v)

Global Alignment with Local

Correlation(2)
Goal: Find the parametric transformation
p, which maximizes the sum of all
normalized correlation values. global
similarity M(p)

( x, y )
M ( p) x. y i Si (u ( x, y; p ), v( x, y; p))
x , y i S

( x, y )
i

(u ( x, y; p ))

Solving for M(p)

Newtons method is used to solve for M(p)

T T

M ( p ) M ( p0 ) ( p M ( p0 )) p p H M ( p0 ) p
*

p ( H M ( p0 )) 1 p M ( p0 )

p M ( p ) x , y ,i p S i (u ) x , y ,i ( X T uS i (u ))

T
H M ( p ) x , y ,i ( X H Si (u ) X )

The quadratic approximation of M around p is

obtained by combining the quadratic
approximations of each of the local correlation
surfaces S around local displacement.

Steps of the Algorithm

Construct a Laplacian resolution pyramid
Compute a local normalized-correlation
surface around a given displacement
Compute the parametric refinement
Update p
Start over(process terminates at the
highest resolution level of the last image)

Outlier Rejection
Due to the different modalities of the
sensors, the number of outliers may be very
large
1. Accept pixels based on concavity of the
correlation surface
2. Weigh the contribution of a pixel by det|
H(u)|

Results

EO image

Composite before Alignment

IR image

IR image

Composite after Alignment

Acknowledgements
I would like to thank Compaq for making
awful computers
Professor Belongie for reviewing the slides