Image Stitching

Shangliang Jiang
Kate Harrison
What is image stitching?
What is image stitching?
 Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
 Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV = 200 x 135°
 Introduction
• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
– Human FOV = 200 x 135°
– Panoramic Mosaic = 360 x 180°
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images

• 2D Rotations (, )
– Ordering  matching images
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images

• 2D Rotations (, )
– Ordering  matching images
 Recognizing Panoramas

• 1D Rotations ()
– Ordering  matching images

• 2D Rotations (, )
– Ordering  matching images
Recognizing Panoramas
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 SIFT Features
• SIFT features are…
– Geometrically invariant to similarity transforms,
• some robustness to affine change
– Photometrically invariant to affine changes in
intensity
 Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Nearest Neighbor Matching
• Find k nearest neighbors for each feature
– k  number of overlapping images (we use k = 4)
• Use k-d tree
– k-d tree recursively bi-partitions data at mean in the
dimension of maximum variance
– Approximate nearest neighbors found in O(nlogn)
 Overview
• Feature Matching
– SIFT Features
– Nearest Neighbor Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
RANSAC for Homography
 Overview
• Feature Matching
• Image Matching
– Random Sample Consensus (RANSAC) for
Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
Probabilistic model for verification
Finding the panoramas
Finding the panoramas
Finding the panoramas
 Overview
• Feature Matching
• Image Matching
– RANSAC for Homography
– Probabilistic model for verification
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
 Bundle Adjustment
• New images initialised with rotation, focal
length of best matching image
 Bundle Adjustment
• New images initialised with rotation, focal
length of best matching image
 Error function
• Sum of squared projection errors

– n = #images
– I(i) = set of image matches to image i
– F(i, j) = set of feature matches between images i,j
– rijk = residual of kth feature match between images i,j

• Robust error function

 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
– Error function
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
Blending
Gain compensation
 How do we blend?

Linear blending Multi-band blending

 Multi-band Blending
• Burt & Adelson 1983
– Blend frequency bands over range  
 2-band Blending

Low frequency ( > 2 pixels)

High frequency ( < 2 pixels)

 3-band blending
Band 1: high frequencies
 3-band blending
Band 2: mid-range frequencies
 3-band blending
Band 3: low frequencies
 Panorama straightening

Heuristic: people tend to shoot pictures in a certain way

 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
 Overview
• Feature Matching
• Image Matching
• Bundle Adjustment
• Image Compositing
• Conclusions
Conclusion
Algorithm
 AutoStitch program

AutoStitch.net
 Open questions
• Advanced camera modeling
– radial distortion, camera motion, scene motion,
vignetting, exposure, high dynamic range, flash …
• Full 3D case – recognizing 3D objects/scenes
in unordered datasets
 Credits
• Automatic Panoramic Image Stitching Using
Invariant Features, 2007
– Matthew Brown and David G. Lowe (Uni. of British
Columbia)
• Recognising Panoramas, 2003
– Matthew Brown and David G. Lowe (Uni. of British
Columbia)
– 2003
– Thanks for the slides!
• Image Alignment and Stitching: A Tutorial,
2006
– Richard Szeliski (Microsoft)
Questions?