Image Processing,

Retrieval, and Analysis (I)

Prof. Christian Bauckhage
Outline
Lecture 22

Recap

Mesh Warping
Mappings between Quadrilaterals

Scanline Mesh Warping

Separability
Applications

Summary
Recap

bilinear mappings between quadrilaterals

x11

xb xd

1 x
x10
x01
v xa
u
xc

x00
u 1
Recap

bilinear mappings
I solution:

x =uv(x11 + x00 − x10 − x01 )+

u(x10 − x00 ) + v(x01 − x00 ) + x00
Recap

bilinear mappings
I solution:

x =uv(x11 + x00 − x10 − x01 )+

u(x10 − x00 ) + v(x01 − x00 ) + x00

I in matrix form:
 
    uv
x a b c d u

=
y e f g h  v
1
Recap

bilinear mappings
I for the coefficients a, b, . . . , h we have

a = x11 + x00 − x10 − x01

b = x10 − x00
c = x01 − x00
d = x00
e = y11 + y00 − y10 − y01
f = y10 − y00
g = y01 − y00
h = y00
Recap

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T becomes an affine map once a = e = 0
Recap

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T becomes an affine map once a = e = 0
I horizontal and vertical lines are mapped onto lines
Recap

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T becomes an affine map once a = e = 0
I horizontal and vertical lines are mapped onto lines
I ratios of distance between equidistant points are preserved
Recap

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T becomes an affine map once a = e = 0
I horizontal and vertical lines are mapped onto lines
I ratios of distance between equidistant points are preserved
I diagonal lines are mapped onto curves
Recap

properties of bilinear mappings

original bilinear warp

Recap

properties of bilinear mappings

I the composition of two bilinear maps is a biquadratic map
Recap

properties of bilinear mappings

I the composition of two bilinear maps is a biquadratic map
I the inverse mapping T −1 , too, is not bilinear; e.g. for
x = auv + bu + cv + d
we find
r 
B B 2 C
u=− ± −
2A 2A A
where
A = af − be
B = ex − ay − de + cf − bg + ah
C = gy − cy + ch − dg
Recap

properties of bilinear mappings

I the composition of two bilinear maps is a biquadratic map
I the inverse mapping T −1 , too, is not bilinear; e.g. for
x = auv + bu + cv + d
we find
r 
B B 2 C
u=− ± −
2A 2A A
where
A = af − be
B = ex − ay − de + cf − bg + ah
C = gy − cy + ch − dg

I i.e. T −1 is not even unique!

Recap

bilinear mappings in image processing

I bilinear mappings T are defined by a set of 8 parameters
{a, b, . . . , h}
Recap

bilinear mappings in image processing

I bilinear mappings T are defined by a set of 8 parameters
{a, b, . . . , h}
I if an image is to be warped bilinearly, these parameters
must be given or have to be determined automatically
Recap

bilinear mappings in image processing

I bilinear mappings T are defined by a set of 8 parameters
{a, b, . . . , h}
I if an image is to be warped bilinearly, these parameters
must be given or have to be determined automatically
I the latter will require 8 equations
Recap

bilinear mappings in image processing

I bilinear mappings T are defined by a set of 8 parameters
{a, b, . . . , h}
I if an image is to be warped bilinearly, these parameters
must be given or have to be determined automatically
I the latter will require 8 equations
I these 8 equations are available, if for 4 points ui in the
source image we know the corresponding 4 points xi in
the destination image
Recap

bilinear mappings in image processing

I we could then derive

x1 = au1 v1 + bu1 + cv1 + d

..
.
x4 = au4 v4 + bu4 + cv4 + d
y1 = eu1 v1 + fu1 + gv1 + h
..
.
y4 = eu4 v4 + fu4 + gv4 + h

I this need for 4 point correspondences is the reason why

we consider mappings between arbitrary quadrilaterals!
Mesh Warping

projective mappings between quadrilaterals

I two quadrilaterals in the plane
Mesh Warping

projective mappings between quadrilaterals

I two quadrilaterals in space
Mesh Warping

projective mappings
I two operations
Mesh Warping

projective mappings
I two operations
I first, a spatial transformation
    0
a b c u u
d e f   v  =  v 0 
g h 1 1 w0
Mesh Warping

projective mappings
I two operations
I first, a spatial transformation
    0
a b c u u
d e f   v  =  v 0 
g h 1 1 w0

I second, a projection

u0 au + bv + c
x= 0
=
w gu + hv + 1
v0 du + ev + f
y= 0 =
w gu + hv + 1
Mesh Warping

projective mappings
I w.l.o.g., entry i in the transformation matrix can be set to 1
Mesh Warping

projective mappings
I w.l.o.g., entry i in the transformation matrix can be set to 1

I choosing another value for i is tantamount to scaling

along the z-axis
Mesh Warping

projective mappings
I w.l.o.g., entry i in the transformation matrix can be set to 1

I choosing another value for i is tantamount to scaling

along the z-axis

I which will be lost anyway in the subsequent projection

Mesh Warping

properties of projective mappings

I projective transformations have the following
characteristics
Mesh Warping

properties of projective mappings

I projective transformations have the following
characteristics
I lines are mapped onto lines
Mesh Warping

properties of projective mappings

I projective transformations have the following
characteristics
I lines are mapped onto lines
I distances between points on
a line are not preserved
but warped perspectively
Mesh Warping

properties of projective mappings

original projective warp

Mesh Warping

projective mappings in image processing

I given 4 pairs of corresponding points

xi = (xi , yi ) ⇔ ui = (ui , vi )
in two images, one can determine the
perspective transformation T between
the images
Mesh Warping

projective mappings in image processing

I given 4 pairs of corresponding points

xi = (xi , yi ) ⇔ ui = (ui , vi )
in two images, one can determine the
perspective transformation T between
the images
I simply consider
aui + bvi + c
xi =
gui + hvi + 1
dui + evi + f
yi =
gui + hvi + 1
⇒xi = ui a + vi b + c − ui xi g − vi yi h
yi = ui d + vi e + f − ui xi g − vi yi h
Mesh Warping

projective mappings in image processing

I in matrix notation, this reads
    
u1 v1 1 0 0 0 −u1 x1 −v1 x1 a x1
u2 v2 1 0 0 0 −u2 x2 −v2 x2   b   x2 
    
u3 v3 1 0 0 0 −u3 x3 −v3 x3 
  c   x3 
   

u4 v4 1 0 0 0 −u4 x4 −v4 x4 
  d  =  x4 
   

 0 0 0 u1 v1 1 −u1 y1 −v1 y1  
 e   y1 
   

 0 0 0 u2 v2 1 −u1 y2 −v2 y2   f 
  y2 
   

 0 0 0 u3 v3 1 −u3 y3 −v3 y3  g y3 
0 0 0 u4 v4 1 −u4 y4 −v4 y4 h y4
Mesh Warping

projective mappings in image processing

I in matrix notation, this reads
    
u1 v1 1 0 0 0 −u1 x1 −v1 x1 a x1
u2 v2 1 0 0 0 −u2 x2 −v2 x2   b   x2 
    
u3 v3 1 0 0 0 −u3 x3 −v3 x3 
  c   x3 
   

u4 v4 1 0 0 0 −u4 x4 −v4 x4 
  d  =  x4 
   

 0 0 0 u1 v1 1 −u1 y1 −v1 y1  
 e   y1 
   

 0 0 0 u2 v2 1 −u1 y2 −v2 y2   f 
  y2 
   

 0 0 0 u3 v3 1 −u3 y3 −v3 y3  g y3 
0 0 0 u4 v4 1 −u4 y4 −v4 y4 h y4

I so that we can easily solve for a, b, . . . , h

Mesh Warping

note:

I the points must be coplanar but not collinear

Mesh Warping

projective mappings in image processing

I applications of projective mappings can be found in
I texture mapping
I augmented reality
I mosaicing
Mesh Warping

texture mapping and augmented reality

Mesh Warping

image mosaics
I if the perspective mapping between a series of image was
known, the images could be rendered into a single mosaic
image
Mesh Warping

image mosaics
I if the perspective mapping between a series of image was
known, the images could be rendered into a single mosaic
image
I in its most simple form, this requires that the series of
images had been captured by a rotating camera or by
a camera moving along a linear trajectory
Mesh Warping

image mosaics
I if the perspective mapping between a series of image was
known, the images could be rendered into a single mosaic
image
I in its most simple form, this requires that the series of
images had been captured by a rotating camera or by
a camera moving along a linear trajectory
I mosaics could also be computed from image series that
violate this simple prerequisite, however, in these cases,
simple perspective mappings do not allow for an accurate
integration of the individual images
Mesh Warping

image mosaics
Mesh Warping

image mosaics
Mesh Warping

image mosaics
Scanline Mesh Warping

bilinear mappings revisited

I a favorable property of the bilinear mapping is that it
is separable
Scanline Mesh Warping

bilinear mappings revisited

I a favorable property of the bilinear mapping is that it
is separable
I advantages of separability became already apparent
when we studied linear filters
Scanline Mesh Warping

bilinear mappings revisited

I a 2D warp function T with

x = T(u)

i.e. with
   
x Tx (u, v)
=
y Ty (u, v)

is separable, if it can be decomposed into two mappings

each of which operates along one of the coordinate axes
only
Scanline Mesh Warping

bilinear mappings revisited

I i.e. T is separable, if there is a function ∆(u, v) such that
   
x Tx (u, v)
=
y ∆(Tx (u, v), v)
Scanline Mesh Warping

bilinear mappings revisited

I i.e. T is separable, if there is a function ∆(u, v) such that
   
x Tx (u, v)
=
y ∆(Tx (u, v), v)

I i.e. Ty (u, v) = ∆(Tx (u, v), v)

Scanline Mesh Warping

bilinear mappings revisited

I if there is such a function ∆, one may use two steps in
order to map coordinates (u, v) onto coordinates (x, y)
         
u T (u, v) x x x
→ x = → =
v v v ∆(x, v) y
Scanline Mesh Warping

bilinear mappings revisited

I generally, a 2D transformation T is separable, if Tx (u, v) is
(partially) invertible
Scanline Mesh Warping

bilinear mappings revisited

I generally, a 2D transformation T is separable, if Tx (u, v) is
(partially) invertible

I we may then solve for u

Scanline Mesh Warping

bilinear mappings revisited

I generally, a 2D transformation T is separable, if Tx (u, v) is
(partially) invertible

I we may then solve for u

I i.e. we may find u = u(x, v)

Scanline Mesh Warping

bilinear mappings revisited

I generally, a 2D transformation T is separable, if Tx (u, v) is
(partially) invertible

I we may then solve for u

I i.e. we may find u = u(x, v)

I the function ∆ then simply results from

∆ = Ty (u(x, v), v)
Scanline Mesh Warping

bilinear mappings revisited

x11

xb xd

1 x
x10
x01
v xa
u
xc

x00
u 1
Scanline Mesh Warping

bilinear mappings revisited

I the previous figure illustrates that a bilinear mapping
between two quadrilaterals is indeed separable
Scanline Mesh Warping

bilinear mappings revisited

I the previous figure illustrates that a bilinear mapping
between two quadrilaterals is indeed separable

I earlier, we already saw that x can be determined

depending on xa and xb

x = v(xb − xa ) + xa
Scanline Mesh Warping

bilinear mappings revisited

I the previous figure illustrates that a bilinear mapping
between two quadrilaterals is indeed separable

I earlier, we already saw that x can be determined

depending on xa and xb

x = v(xb − xa ) + xa

I some straightforward algebra led to

x =uv(x11 + x00 − x10 − x01 )+

u(x10 − x00 ) + v(x01 − x00 ) + x00
Scanline Mesh Warping

bilinear mappings revisited

I the latter, however, can also be written as
 
x =u v(x11 − x10 ) + x10 − v(x01 − x00 ) − x00 +
v(x01 − x00 ) + x00
Scanline Mesh Warping

bilinear mappings revisited

I the latter, however, can also be written as
 
x =u v(x11 − x10 ) + x10 − v(x01 − x00 ) − x00 +
v(x01 − x00 ) + x00

i.e.

x = u(xd − xc ) + xc
Scanline Mesh Warping

bilinear mappings revisited

I the latter, however, can also be written as
 
x =u v(x11 − x10 ) + x10 − v(x01 − x00 ) − x00 +
v(x01 − x00 ) + x00

i.e.

x = u(xd − xc ) + xc

where

xc = v(x01 − x00 ) + x00

and

xd = v(x11 − x10 ) + x10

Scanline Mesh Warping

bilinear mappings revisited

I we may therefore
I either transform the u component of u first and then
transform the v component
I or transform the v component first and then apply a
transformation along the u direction
in order to obtain (x, y)
Scanline Mesh Warping

mesh warping
I the separability of bilinear maps provides a simple and
efficient way of warping an image locally
Scanline Mesh Warping

mesh warping

meshes cast over an image distorted meshes

Scanline Mesh Warping

mesh warping
I a regular mesh or lattice is defined over an image
Scanline Mesh Warping

mesh warping
I a regular mesh or lattice is defined over an image
I a slightly different quadrilateral structure results from
small displacements of the vertices of the mesh
Scanline Mesh Warping

mesh warping
I a regular mesh or lattice is defined over an image
I a slightly different quadrilateral structure results from
small displacements of the vertices of the mesh
I when shifting vertices of the mesh, we must ensure that

1. lattice points in the corners of an image are not displaced

2. lattice points located on image borders are only displaced
along these borders
3. any displacement preserves the lattice topology, i.e. the
edges of the mesh must not intersect after vertices have
been shifted
Scanline Mesh Warping

mesh warping

g(u, v) ĝ(x, v) g̃(x, y)

Scanline Mesh Warping

approach
### step 1: distortion along the x direction
for v in range(rmax):
for k in range(numGridPointsX):
for l in range(numGridPointsY):
srcColX[l] = srcGridX[l,k]
srcColY[l] = srcGridY[l,k]
dstColX[l] = dstGridX[l,k]
dstColY[l] = dstGridY[l,k]
srcIntersectX[k] = interpol(srcColY, srcColX, numGridPointsY, v)
dstIntersectX[k] = interpol(dstColY, dstColX, numGridPointsY, v)

for x in range(cmax):
u = interpol(dstIntersectX, srcIntersectX, numGridPointsX, x)
gDx[v*cmax,x] = resample(gOrig, u, v)

### step 2: distortion along y-direction

for x in range(cmax):
for l in range(numGridPoints):
for k in range(numGridPointsX):
srcRowX[k] = srcGridX[l][k]
srcRowY[k] = srcGridY[l][k]
dstRowX[k] = dstGridX[l][k]
dstRowY[k] = dstGridY[l][k]
srcIntersectY[l] = interpol(srcRowx, srcRowy, numGridPointsX, x)
dstIntersectY[l] = interpol(dstRowx, dstRowy, numGridPointsX, x)

for y in range(rmax):
v = interpol(dstIntersectY, srcIntersectY, numGridPointsY, y)
gDxDy[y*cmax,x] = resample(gDx, x, v)
Scanline Mesh Warping

mesh warping
I once again

meshes cast over an image distorted meshes

Scanline Mesh Warping

mesh warping
I explanation
I in the first step, an intermediate image gDx is computed
that shows a warped version of the original image where
the warp was applied along the x direction
I to this end, the script iterates over all rows v of the image;
in each iteration, the vertical lines (i.e. the columns) of the
original mesh are stored in srcCol and the vertical lines of
the warped mesh are stored in dstCol; subsequently, the
points of intersection of the vertical mesh lines with the
current row of the image are computed; the x-coordinates of
these points of intersection with image row v are stored in
the arrays srcIntersectX and dstIntersectX,
respectively
Scanline Mesh Warping

mesh warping
I explanation (cont.)
I then, the script iterates over all columns x of the current
row v and, by means of interpolation between the recently
collected points of intersection of the row with the lines of
the original mesh and the lines of the distorted mesh, it is
being determined, which column u in the original image
gOrig corresponds to the current column x
I finally, the color value at coordinate (u,v) is read from
gOrig and stored at coordinate (x,v) in gDx
Scanline Mesh Warping

mesh warping
I explanation (cont.)
I in a second step, the intermediate result gDx is turned into
the desired warped image gDxDy
I this happens in analogy to the first step but the roles of
rows and columns are exchanged
Scanline Mesh Warping

caricatures
I using the technique to create caricatures is straightforward
Scanline Mesh Warping

mesh morphing
I the famous effect of morphing, too, has its roots in mesh
warping
Scanline Mesh Warping

algorithm
define a grid GS to overlay over the source image IS
define a grid GD to overlay over the destination image ID
(corresponding grid points should ideally come to lie over
corresponding image contents)

in order to create the n intermediate images, do the following

for t = 1, . . . , n − 1
create an interpolated grid Gt
GD − GS
Gt = t
n
apply GS and Gt , in order to warp IS into an image ISt
apply GD and Gt , in order to warp ID into an image IDt
create an intermediate image It by blending ISt and IDt
t t
It = (1 − ) ISt + IDt
n n
Scanline Mesh Warping

mesh morphing
Scanline Mesh Warping

mesh morphing
I the second and fourth rows in the figure show the warped
images ISt and IDt , respectively
I the middle row in the figure shows the images IS , It (where
t ∈ {1, 2, 3}) and ID
Scanline Mesh Warping

mesh morphing
I the second and fourth rows in the figure show the warped
images ISt and IDt , respectively
I the middle row in the figure shows the images IS , It (where
t ∈ {1, 2, 3}) and ID

note:

I meanwhile, there are many more and (much) more

elaborate approaches to morphing
I field morphing
I transition control
I multilevel free form deformation, . . .
Summary

we now know about

I perspective mappings and their properties

I applications of perspective warping (augmented reality,

mosaics)

I the fact that bilinear mappings are separable

I the idea of mesh warping and applications (caricatures,

morphing)