Image warping/morphing

Digital
g Visual Effects
Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros
Image warping
Image formation

A
Sampling and quantization
What is an image
• We can think of an image as a function, f: R2R:
– f(x,
f( y)) gives
i th
the intensity
i t it att position
iti ((x, y))
– defined over a rectangle, with a finite range:
• f:
f [a,b]x[c,d]
[ b] [ d]  [0,1]
[0 1]
f
x

• A color image  r(x, y) 

f (x, y)   g (x, y)
 
 b ( x , y ) 
A digital image
• We usually operate on digital (discrete) images:
– Sample
S l the
th 2D space on a regular
l grid
id
– Quantize each sample (round to nearest integer)
• Iff our samples
l are D apart, we can write this
h as:
f[i ,j] = Quantize{ f(i D, j D) }
• The image can now be represented as a matrix
of integer
o tege values
Image warping
image filtering: change range of image
g(x)
( ) = h(f(
h(f(x))
))
h(y)=0.5y+0.5
f g
h
x x

image warping: change domain of image

g(x) = f(h(x))
h(y)=2y
f g
h
x x
Image warping
image filtering: change range of image
g(x)
( ) = h(f(
h(f(x))
))
h(y)=0.5y+0.5
f g
h

image warping: change domain of image

g(x) = f(h(x))
h([x,y])=[x,y/2]
f
h
g
Parametric (global) warping

Examples of parametric warps:

translation rotation aspect

perspective
affine cylindrical
Parametric (global) warping

p = (x,y) p’ = (x
p (x’,y’)
,y )
• Transformation T is a coordinate-changing
machine: p
p’ = T(p)
• What does it mean that T is global?
– Is the same for any point p
– can be described by just a few numbers (parameters)
• Represent
R t T as a matrix:
t i p’’ = M*p
M*
 x' x
 y''  M  y 
   
Scaling
• Scaling a coordinate means multiplying each of
its components by a scalar
• Uniform scaling means this scalar is the same
for all components:
g
f  x  x'  2 x 
 y  y '  2 y 
     

2
Scaling
• Non-uniform scaling: different scalars per
component:
  x    x' 
f      g    
  y    y ' 
 x'  2 x 
 y '  0.5 y 
   

x  2,
y  0.5
Scaling

• Scaling operation: x '  ax

y '  by
• Or, in matrix form:
 x '  a 0  x 
 y '   0 b   y 
    
scaling matrix S
What’s inverse of S?
2-D Rotation
• This is easy to capture in matrix form:
 x '  cos   sin    x 
 y '   sin   cos    y 
    

R
• Even though sin() and cos() are nonlinear to ,
– x’’ is
i a linear
li combination
bi ti off x and dy
– y’ is a linear combination of x and y
• What
Wh iis the
h iinverse transformation?
f i ?
– Rotation by –
1
– For rotation matrices, det(R) = 1 so R  R
1 T
2x2 Matrices
• What types of transformations can be
represented with a 2x2 matrix?

2D Identity?
x'  x  x '   1 0   x 
y'  y  y ' 0 1  y 

2D Scale around (0,0)?

x'  s x * x  x '  s x 0  x
y'  s y * y  y '   0 
s y   y 
  
2x2 Matrices
• What types of transformations can be
represented with a 2x2 matrix?

2D Rotate around (0,0)?

x'  cos  * x  sin
i *y  x' cos  i   x
 sin
y '  sin  * x  cos  * y  y '   sin  cos    y 
  

2D Shear?
x '  x  shx * y  x '  1 shx   x 
y '  shy * x  y  y '   sh 
1   y 
   y
2x2 Matrices
• What types of transformations can be
represented with a 2x2 matrix?

2D Mirror about Y axis?

x'   x  x '     1 0  x 
y'  y  y '  0 1  y 

2D Mirror over (0,0)?

x'   x  x'   1 0   x 
y'   y  y '  0  1  y 
All 2D Linear Transformations
• Linear transformations are combinations of …
– Scale,
Scale
– Rotation,
– Shear and
Shear,
– Mirror
• Properties of linear transformations:
– Origin maps to origin
– Lines map to lines
– Parallel lines remain parallel
– Ratios are ppreserved
 xx'' a b   x 
 y '   c d   y 
– Closed under composition

    
2x2 Matrices
• What types of transformations can not be
represented with a 2x2 matrix?

2D Translation?
x'  x  t x
NO!
y'  y  t y

Only linear 2D transformations

can be
b represented
t d with
ith a 2
2x2
2 matrix
ti
Translation
• Example of translation
Homogeneous Coordinates

 x '  1 0 t x   x   x  t x 
 y '   0 1 t   y    y  t 
   y    y

 1  0 0 1   1   1 

tx = 2
ty = 1
Affine Transformations
• Affine transformations are combinations of …
– Linear
Li transformations,
f i and
d
– Translations
• Properties of affine transformations:
– Origin does not necessarily map to origin
– Lines map to lines
– Parallel lines remain parallel
– Ratios are preserved
– Closed under composition  x '  a b c  x 
– Models change of basis  y '  d e f  y 
 w 0 0 1   w
  
Projective Transformations
• Projective transformations …
– Affine transformations,
transformations and
– Projective warps
• Properties
P ti off projective
j ti ttransformations:
f ti
– Origin does not necessarily map to origin
– Lines map to lines
l
– Parallel lines do not necessarily remain parallel
– Ratios are not preserved
– Closed under composition  x'  a b c  x 
– Models change of basis  y '    d e f  y 
 w'  g h i   w
Image warping
• Given a coordinate transform x’ = T(x) and a
source image I(x),
I(x) how do we compute a
transformed image I’(x’) = I(T(x))?

T(x)
(x)

x x’
I(x) II’(x’)
(x )
Forward warping
• Send each pixel I(x) to its corresponding
location x’ = T(x) in I’(x’)

T(x)
(x)

x x’
I(x) II’(x’)
(x )
Forward warping
fwarp(I, I’, T)
{
for (y=0; y<I.height; y++)
for (x=0; x<I
x<I.width;
width; x++) {
(x’,y’)=T(x,y);
I’(x’ y’) I(x y);
I’(x’,y’)=I(x,y);
}
}
I I’
T

x’
Forward warping
Some destination may
not be covered

Many source pixels could map

to the same destination
Forward warping
• Send each pixel I(x) to its corresponding
location x’ = T(x) in I’(x’)
• What if p
pixel lands “between” two ppixels?
• Will be there holes?
• Answer: add “contribution”
contribution to several pixels
pixels,
normalize later (splatting)

h(x)
(x)

x x’
f(x) g(x’))
g(x
Forward warping
fwarp(I, I’, T)
{
for (y=0; y<I.height; y++)
for (x=0; x<I
x<I.width;
width; x++) {
(x’,y’)=T(x,y);
Splatting(I’ x’ y’ I(x y) kernel);
Splatting(I’,x’,y’,I(x,y),kernel);
}
}
I I’
T

x’
Inverse warping
• Get each pixel I’(x’) from its corresponding
location x = T-11(x’) in I(x)

T-1(x
(x’))

x x’
I(x) II’(x’)
(x )
Inverse warping
iwarp(I, I’, T)
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’
x<I’.width;
width; x++) {
(x,y)=T-1(x’,y’);
I’(x’ y’) I(x y);
I’(x’,y’)=I(x,y);
}
} T-1
I I’

x
x’
Inverse warping
• Get each pixel I’(x’) from its corresponding
location x = T-11(x’) in I(x)

• What if pixel comes from “between” two pixels?

• Answer: resample color value from
interpolated (prefiltered) source image

x x’
f(x) g(x’))
g(x
Inverse warping
iwarp(I, I’, T)
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’
x<I’.width;
width; x++) {
(x,y)=T-1(x’,y’);
I’(x’ y’) Reconstruct(I x y kernel);
I’(x’,y’)=Reconstruct(I,x,y,kernel);
}
} T-1
I I’

x
x’
Inverse warping
• No hole, but must resample
• What
Wh valuel should
h ld you take
k ffor non-integer
i
coordinate? Closest one?
Inverse warping
• It could cause aliasing
Reconstruction
• Reconstruction generates an approximation to
the original function
function. Error is called aliasing
aliasing.
sampling reconstruction
sample
l value
l

sample
l position
i i
Reconstruction
• Computed weighted sum of pixel neighborhood;
output is weighted average of input,
input where
weights are normalized values of filter kernel k
p
 k (q )q
i i i

 k (q )
i i
color=0;
weights=0;
for all q
q’s dist < width
p width d = dist(p, q);
w = kernel(d);
d color += w*q.color;
q weights += w;
p Color = color/weights;
p.Color
Triangle filter
Gaussian filter
Sampling
band limited
Reconstruction

The reconstructed function is obtained by interpolating

among the samples in some manner
Reconstruction (interpolation)
• Possible reconstruction filters (kernels):
– nearestt neighbor
i hb
– bilinear
– bi bi
bicubic
– sinc (optimal reconstruction)
Bilinear interpolation (triangle filter)
• A simple method for resampling images
Non-parametric image warping
• Specify a more detailed warp function
• Splines,
S li meshes,
h optical
i l fl
flow ((per-pixel
i l motion)
i )
Non-parametric image warping
• Mappings implied by correspondences
• Inverse
I warping
i

P’
?
Non-parametric image warping
P '  wA A' wB B' wC C '
P  wA A  wB B  wC C
Barycentric coordinate

P P’
Barycentric coordinates

P  t1 A1  t 2 A2  t3 A3
t1  t 2  t3  1
Non-parametric image warping

P'  wA A' wB B' wC C '

P  wA A  wB B  wC C
Barycentric coordinate
Non-parametric image warping
 r 2
 (r )  e 1
Gaussian P 
K
k Xi ( P' )X i
thin plate  (r )  r log(r )
2 i
spline radial basis function
Image warping
• Warping is a useful operation for mosaics, video
matching view interpolation and so on
matching, on.
An application of image warping:
face beautification
Data-driven facial beautification
Facial beautification
Facial beautification
Facial beautification
Training set
• Face images
– 92 young C
Caucasian
i ffemalel
– 33 young Caucasian male
Feature extraction
Feature extraction
• Extract 84 feature points by BTSM
• Delaunay
D l triangulation
i l i -> 234D distance
di vector
(normalized by the square root of face area)

scatter
tt plot
l t ffor 234D vector
BTSM
all training faces
Beautification engine
Support vector regression (SVR)
• Similar concept to SVM, but for regression
• RBF kernels
k l
• fb(v)
Beautification process
• Given the normalized distance vector v,
generate a nearby vector vv’ so that
fb(v’) > fb(v)

• Two options
p
– KNN-based
– SVR-based
SVR based
KNN-based beautification

5.1

4.3

3.1
31
v' v

45
4.5

4.6
5.3
SVR-based beautification
• Directly use fb to seek v’

• Use standard no-derivative direction set

method for minimization
• Features were reduced to 35D by PCA
 SVR-based beautification
• Problems: it sometimes yields distance vectors
corresponding to invalid human face
• Solution: add log-likelihood term (LP)

• LP is approximated
pp byy modelingg face space
p as a
multivariate Gaussian distribution ’s i-th
component

u’s projection i-th eigenvalue

in PCA space
PCA

λ2

λ1
Embedding and warping
Distance embedding
• Convert modified distance vector v’ to a new
face landmark

1 if i and
djbbelong
l tto diff
differentt ffacial
i l ffeatures
t
10 otherwise

• A graph drawing problem referred to as a stress

minimization
i i i ti problem,
bl solved
l db by LM algorithm
l ith
for non-linear minimization
Distance embedding
• Post processing to enforce similarity transform
for features on eyes by minimizing
Original K=3 K=5 SVR
Results (in training set)
User study
Results (not in training set)
By parts

eyes

mouth

full
Different degrees

50% 100%
Facial collage
Image morphing
Image morphing
• The goal is to synthesize a fluid transformation
from one image to another.
another
• Cross dissolving is a common transition between
cuts, but it is not good for morphing because of
the ghosting effects.

i g #1
image di l i g
dissolving image #2
Artifacts of cross-dissolving

http://www.salavon.com/
Image morphing
• Why ghosting?
• Morphing
M hi = warpingi + cross-dissolving
di l i

shape color
(geometric) (photometric)
Image morphing
image #1 cross-dissolving image #2

warp morphing warp

Morphing sequence
Face averaging by morphing

average faces
Image morphing
create a morphing sequence: for each time t
1 C
1. Create
t an iintermediate
t di t warping
i fifield
ld (b
(by
interpolation)
2 Warp both images towards it
2.
3. Cross-dissolve the colors in the newly warped
images

t=0 t=0.33 t=1

An ideal example (in 2004)

t=0 morphing
t=0.25
t=0.5
t=0.75 t=1
An ideal example

t=0 middle face (t=0.5) t=1

Warp specification (mesh warping)
• How can we specify the warp?
1 SSpecify
1. if corresponding
di spline
li control
t l points
i t
interpolate to a complete warping function

easy to implement,
implement but less expressive
Warp specification
• How can we specify the warp
2 SSpecify
2. if corresponding
di points
i t
• interpolate to a complete warping function
Solution: convert to mesh warping

1 D
1. Define
fi a triangular
ti l mesh h over th
the points
i t
– Same mesh in both images!
– Now
N we hhave triangle-to-triangle
i l i l correspondences
d
2. Warp each triangle separately from source to destination
– How do we warp a triangle?
– 3 points = affine warp!
– Just like texture mapping
Warp specification (field warping)
• How can we specify the warp?
3 Specify
3. S if corresponding
di vectors
t
• interpolate to a complete warping function
• The Beier & Neely Algorithm
Beier&Neely (SIGGRAPH 1992)
• Single line-pair PQ to P’Q’:
Algorithm (single line-pair)
• For each X in the destination image:
1. Fi
1 Find
d the
th corresponding
di u,v
2. Find X’ in the source image for that u,v
3 d
3. ti ti I (X) = sourceImage(X’)
destinationImage(X) I (X’)
• Examples:

Affine transformation
Multiple Lines

Di  X i'  X i

length = length of the line segment,

dist = distance to line segment
The influence of a, p, b. The same as the average of Xi’
Full Algorithm
Resulting warp
Comparison to mesh morphing
• Pros: more expressive
• Cons:
C speed
d and
d controll
Warp interpolation
• How do we create an intermediate warp at
time t?
– linear interpolation for line end-points
– But,
B t a line
li rotating
t ti 180 degrees
d will
ill b
become 0
length in the middle
– One solution is to interpolate line mid-point
mid point and
orientation angle
t=0

t=1
Animation
Animated sequences
• Specify keyframes and interpolate the lines for
the inbetween frames
• Require a lot of tweaking
Results

Michael Jackson’s MTV “Black or White”

Multi-source morphing
Multi-source morphing
Woman in arts
References
• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis,
SIGGRAPH 1992, pp35-42.
• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered
Data Interpolation, IEEE Computer Graphics and Applications,
March 1995,
1995 pp37-43
pp37 43.
• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image
Metamorphosis Using Snakes and Free-Form Deformations,
SIGGRAPH 1995.
1995
• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing
among g multiple
p images,
g , IEEE Computer
p Graphics
p and Applications,
pp ,
Vol. 18, No. 1, 1998, pp58-71.
• Peinsheng Gao, Thomas Sederberg, A work minimization approach
to image morphing,
morphing The Visual Computer,
Computer 1998,
1998 pp390-400.
pp390 400
• George Wolberg, Image morphing: a survey, The Visual Computer,
1998, pp360-372.
• Data-Driven Enhancement of Facial Attractiveness, SIGGRAPH 2008