Pinhole camera model

((X,Y,Z)
, , )
P
origin
p
(x,y)

principal point
(optical
center)

• The coordinate system

– We
W willill use the
th pin-hole
i h l model
d l as an approximation
i ti
– Put the optical center (Center Of Projection) at the origin
– Put the image plane (Projection Plane) in front of the COP (Why?)
Pinhole camera model

fX
x=
Z
fY
principal
p p y=
point Z

⎛ x ⎞ ⎛ fX ⎞ ⎡ f 0 0⎤ ⎛ X ⎞
⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎜ ⎟
⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 f 0⎥ ⎜ Y ⎟
⎜1⎟ ⎜ Z ⎟ ⎢0 1⎥⎦⎝⎜ Z ⎟⎠
⎝ ⎠ ⎝ ⎠ ⎣ 0
Pinhole camera model

((X,Y,Z)
, , )
P
origin
p
(x,y)

principal point
(optical y
center)
x
⎛ x ⎞ ⎛ fX ⎞ ⎡ f 0 0⎤ ⎛ X ⎞
⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎜ ⎟
⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 f 0⎥ ⎜ Y ⎟
⎜1⎟ ⎜ Z ⎟ ⎢0 1⎦⎥⎜⎝ Z ⎟⎠
⎝ ⎠ ⎝ ⎠ ⎣ 0
Pinhole camera model

((X,Y,Z)
, , )
P
origin
p
(x,y)

principal point
(optical y
center)
x
⎛ x ⎞ ⎛ fX ⎞ ⎡ f 0 x0 ⎤⎛ X ⎞
⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎜ ⎟
⎜ y ⎟ ~ ⎜ fY ⎟ = ⎢ 0 f y0 ⎥ ⎜ Y ⎟
⎜1⎟ ⎜ Z ⎟ ⎢0 1 ⎦⎥⎜⎝ Z ⎟⎠
⎝ ⎠ ⎝ ⎠ ⎣ 0
Intrinsic matrix

Is this form of K good enough? ⎡f 0 xc ⎤

K = ⎢⎢ 0 f yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦

• non-square pixels (digital video)

⎡ fx 0 xc ⎤
K = ⎢⎢ 0 fy yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦
Intrinsic matrix

Is this form of K good enough? ⎡f 0 xc ⎤

K = ⎢⎢ 0 f yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦

• non-square pixels (digital video)

• skew ⎡f s xc ⎤
x

K = ⎢⎢ 0 fy yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦
Intrinsic matrix

Is this form of K good enough? ⎡f 0 xc ⎤

K = ⎢⎢ 0 f yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦

• non-square pixels (digital video)

• skew ⎡ fx s xc ⎤
• radial distortion K=⎢0 ⎢ fy yc ⎥⎥
⎢⎣ 0 0 1 ⎥⎦
Distortion

• Radial distortion of the image

g
– Caused by imperfect lenses
– Deviations are most noticeable for rays that pass
through the edge of the lens
Barrel Distortion

No distortion

Wide Angle Lens

Barrel
Pin Cushion Distortion

No distortion

Telephoto lens
Pin cushion
Modeling distortion
Distortion-Free: With Distortion:
X
fX 1. Project (X, Y, Z) xn =
x= to “normalized”
normalized
Z
Z image coordinates Y
yn =
Z
fY
y=
Z r 2 = xn2 + yn2
2. Apply radial distortion (
xd = xn 1 + κ 1r 2 + κ 2 r 4 )
yd = y (1 + κ r
n 1
2
+κ r )
2
4

3. Apply
pp y focal length
g x = fxd + xc
translate image center y = fyd + yc

• To model lens distortion

– Use above projection operation instead of
standard projection matrix multiplication
Camera rotation and translation

⎛ X '⎞ ⎛X⎞
⎜ ⎟ ⎜ ⎟
⎜ Y ' ⎟ = R 3×3 ⎜ Y ⎟ + t
⎜ Z'⎟ ⎜Z⎟
⎝ ⎠ ⎝ ⎠
⎛ x⎞ ⎡ f 0 xc ⎤
⎛X⎞
⎜ ⎟ x ~ K [R t ]X
⎜ ⎟ ⎢
yc ⎥[R t ]⎜ ⎟
⎥ ⎜Y ⎟
⎜ y⎟ ~ ⎢0 f
⎜1⎟ ⎢0 Z
⎝ ⎠ ⎣ 0 1 ⎥⎦ ⎜ ⎟ extrinsic matrix
⎜1⎟
⎝ ⎠
Two kinds of parameters

• internal or intrinsic parameters: focal length,

optical center, skew
• external or extrinsic (p
(pose):
) rotation and
translation:

⎛X⎞ ⎛X⎞
⎛ x⎞ ⎡ f 0 xc ⎤ ⎜ ⎟ ⎡a b c d ⎤⎜ ⎟
⎜ ⎟ ⎢
yc ⎥[R t ]⎜ ⎟ = ⎢ e
⎥ ⎜Y ⎟ ⎢ ⎥ ⎜Y ⎟
⎜ y⎟ ~ ⎢0 f f g h ⎥⎜ ⎟
⎜1⎟ ⎢0 Z Z
⎝ ⎠ ⎣ 0 1 ⎥⎦ ⎜ ⎟ ⎢⎣ i j k l ⎥⎦⎜⎜ ⎟⎟
⎜1⎟
⎝ ⎠ ⎝1⎠
Other projection models
Orthographic projection

• Special case of perspective projection

– Distance from the COP to the PP is infinite

Image World

– Also called “parallel projection”: (x, y, z) → (x, y)

Other types of projections

• Scaled orthographic
– Also called “weak perspective”

• Affine projection
– Also called “paraperspective”
Fun with perspective
Perspective cues
Perspective cues
Fun with perspective

Ames room
Forced perspective in LOTR

Elijah Wood: 5' 6" (1.68 m) Ian McKellen 5' 11" (1.80 m)
Camera calibration
Camera calibration

⎛X⎞ ⎛X⎞
⎛ x⎞ ⎡ f 0 xc ⎤ ⎜ ⎟ ⎡a b c d ⎤⎜ ⎟
⎜ ⎟ ⎢
yc ⎥[R t ]⎜ ⎟ = ⎢ e
⎥ ⎜Y ⎟ ⎢ ⎥ ⎜Y ⎟
⎜ y⎟ ~ ⎢0 f f g h ⎥⎜ ⎟
⎜1⎟ ⎢0 Z Z
⎝ ⎠ ⎣ 0 1 ⎥⎦ ⎜ ⎟ ⎢⎣ i j k l ⎥⎦⎜⎜ ⎟⎟
⎜1⎟
⎝ ⎠ ⎝1⎠
• Estimate both intrinsic and extrinsic
parameters
• Mainly, two categories:
1. Using objects with known geometry as
reference
2. Self calibration (structure from motion)
Camera calibration approaches

• Directly estimate 11
unknowns in the M matrix
using known 3D points
((Xi,Yi,Zi)
, , ) and measured
feature positions (ui,vi)

x ~ K [R t ]X = MX
Linear regression
Linear regression

⎡m00 ⎤
⎢m ⎥
⎢ 01 ⎥
⎢m02 ⎥
⎢ ⎥
⎢ m03 ⎥
⎢ m10 ⎥
⎡Xi Yi Zi 1 0 0 0 0 − ui X i − uiYi − ui Z i ⎤ ⎢ ⎥ ⎡ui ⎤
⎢0 ⎥ ⎢ m11 ⎥ = ⎢ ⎥
⎣ 0 0 0 Xi Yi Zi 1 − vi X i − viYi − viYi ⎦ ⎢ ⎥ ⎣ vi ⎦
m
⎢ 12 ⎥
⎢ m13 ⎥
⎢m ⎥
⎢ 20 ⎥
⎢ m21 ⎥
⎢ ⎥
⎣m22 ⎦
Linear regression
⎡ m00 ⎤
⎢m ⎥
⎢ 01 ⎥
⎢ m02 ⎥
⎢ ⎥
⎡ X 1 Y1 Z1 1 0 0 0 0 − u1 X 1 − u1Y1 − u1Z1 ⎤ ⎢ m03 ⎥
⎢ 0 0 0 0 X 1 Y1 Z1 1 − v1 X 1 − v1Y1 − v1Y1 ⎥ ⎢ m10 ⎥
⎢ ⎥ ⎢ ⎥ ⎡ui ⎤
⎢ M ⎥ ⎢ m11 ⎥ = ⎢ v ⎥
⎢ X N YN Z N 1 0 0 0 0 − uN X N − u N YN − u N Z N ⎥ ⎢ ⎥ ⎣ i ⎦
m12
⎢ 0 0 0 0 X N YN Z N 1 − vN X N − v N YN − v N YN ⎥ ⎢ ⎥
⎣ ⎦ ⎢ m13 ⎥
⎢m ⎥
⎢ 20 ⎥
⎢ m21 ⎥
⎢ ⎥
⎣m22 ⎦

Solve for Projection Matrix M using least

least-
square techniques
Normal equation (Geometric Interpretation)
Given an overdetermined system
Ax = b
the normal equation is that which minimizes
the sum of the square differences between
left and right sides
min Ax − b
2

⇔ A ( Ax − b) = 0
T

⇔x= A A A b
T
(
−1 T
)
Normal equation (Differential Interpretation)

E (x) = Ax − b
2

⎡ a11 ... a1m ⎤ ⎡ b1 ⎤

⎢ : ⎥ ⎢ ⎥
⎢ : ⎥ ⎡ x 1 ⎤ ⎢ ⎥:
⎢ : ⎢ ⎥
: ⎥⎢ : ⎥ = ⎢ : ⎥
⎢ ⎥ ⎢ ⎥
⎢ : : ⎥ ⎢
⎣ x m
⎥
⎦ ⎢ ⎥:
⎢⎣an1 ... anm ⎥⎦ ⎢⎣bn ⎥⎦
nxm, n equations,
i m variables
i bl
Normal equation

E (x) = Ax − b
2

= (Ax − b ) (Ax − b )
T

= (x T
A −b
T T
)(Ax − b )
Carl Friedrich Gauss
= x A Ax − b Ax − x A b + b b
T T T T T T

(
= x A Ax − A b x − A b x + b b
T T T
)
T
( T
)
T T

∂E
= 2 A Ax − 2 A b
T T

∂x Who
h invented
i d Least Square?
S ?
Nonlinear optimization

• A probabilistic view of least square

• Feature measurement equations

• Likelihood of M given {(ui,vi)}

 Optimal estimation

• Log likelihood of M given {(ui,vi)}

2 2
N
1 ⎛ m00 X i + m01Yi + m02 Z i + m03 ⎞ 1 ⎛ m10 X i + m11Yi + m12 Z i + m13 ⎞
= ∑ 2 ⎜⎜ ui − ⎟⎟ + 2 ⎜⎜ vi − ⎟⎟
i =1 σ i ⎝ m20 X i + m21Yi + m22 Z i + 1 ⎠ σ i ⎝ m20 X i + m21Yi + m22 Z i + 1 ⎠

• It is a least square problem (but not

necessarily linear least square)
• How do we minimize C?
Nonlinear least square methods
Least square fitting

number of data points

number of parameters
Nonlinear least square fitting
Function minimization
Least square is related to function minimization.

It is very hard to solve in general. Here, we only

consider a simpler
p p problem of finding
g local minimum.
Function minimization
Quadratic functions
Approximate the function with
a quadratic function within
a small neighborhood
Function minimization
Recap: Calibration

• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui
(ui,vi)
vi)

x ~ K [R t ]X = MX

Camera Model:
Recap: Calibration

• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui
(ui,vi)
vi)

x ~ K [R t ]X = MX

Linear Approach:
Recap: Calibration

• Directly estimate 11
unknowns in the M matrix
using known 3D points
(Xi,Yi,Zi) and measured
feature positions (ui
(ui,vi)
vi)

x ~ K [R t ]X = MX

NonLinear Approach:
2 2
N
⎛ m00 X i + m01Yi + m02 Z i + m03 ⎞ ⎛ m10 X i + m11Yi + m12 Z i + m13 ⎞
∑ ⎜⎜ ui −
i =1 ⎝
⎟⎟ + ⎜⎜ vi −
m20 X i + m21Yi + m22 Z i + 1 ⎠ ⎝ m20 X i + m21Yi + m22 Z i + 1 ⎠
⎟⎟
Practical Issue

is hard to make and the 3D feature

positions are difficult to measure!
A popular calibration tool
Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage
• Only requires a plane
• Don
Don’tt have to know positions/orientations
• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
– Matlab version by Jean-Yves
Jean Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
Step 1: data acquisition
Step 2: specify corner order
Step 3: corner extraction
Step 3: corner extraction
Step 4: minimize projection error
Step 4: camera calibration
Step 4: camera calibration
Step 5: refinement