Panorama Stitching

Part II (Stitch multiple images)

Image Processing - OpenCV, Python & C++

By: Rahul Kedia

Source Code:
https://github.com/KEDIARAHUL135/PanoramaStitching
P2.git

1
Table of contents

Overview 3

Approach 3

Setting up initial code 4

Cylindrical Projection and Unroll 6

Stitching the images 11

Future Scopes 13

2
Overview

Having learned the method of stitching 2 images together in the project Panorama Stitching
Part 1, in this project we’ll continue with that method to stitch multiple images together to
create a wide view panorama.

Approach
Knowing how to stitch 2 images together from part 1 of this project, we will use that method
to stitch multiple images together. We will treat the next image to be stitched as the
Secondary Image and the panoramic image formed by stitching the previous images as the
Base Image. In this way, we will use the code to stitch 2 images together in a loop giving
new images one by one to be stitched on the previously stitched images.

Is that all we need to do? The answer is no. As we try to increase the number of images to
be stitched, we will face heavy distortion of images on the sides on giving incorrect results.
Why this happens will be discussed in the respective chapter. To solve this problem, we will
first project the images on the cylinder, then unroll them, and then finally stitch them
together.

Another thing that we will need to correct in the previous project is the method used to
overlap the images. The method used in that project is suitable for 2 images but it will break
if we try to overlap more images on that.

3
Setting up initial code
Let us first have a look at the code to read the images to be stitched from a folder in order
and pass these images one by one for stitching.

NOTE: Inside the folder, the images are stored with names as “1.jpg”, “2.jpg”, …., and so on.
Also, it is noted that the numbered image should have something in common (overlapping
region) with any of the previous images.

if __name__ == "__main__":
# Reading images.
Images = ReadImage("InputImages/Field")

BaseImage, _, _ = ProjectOntoCylinder(Images[0])
for i in range(1, len(Images)):
StitchedImage = StitchImages(BaseImage, Images[i])

BaseImage = StitchedImage.copy()

cv2.imwrite("Stitched_Panorama.png", BaseImage)

This is the main condition from where the code starts. The images are first read using the
function ReadImage() and then passed one by one to the function StitchImages() along
with the Stitched image.

Let’s see how the ReadImage() function reads the images. Everything else will be discussed
in the respective chapters.

def ReadImage(ImageFolderPath):
# Input Images will be stored in this list.
Images = []

# Checking if path is of folder.

# If path is of a folder contaning images.
if os.path.isdir(ImageFolderPath):
ImageNames = os.listdir(ImageFolderPath)
ImageNames_Split = [[int(os.path.splitext(
os.path.basename(ImageName))[0]),
ImageName] for ImageName in ImageNames]
ImageNames_Split = sorted(ImageNames_Split, key=lambda x:x[0])
ImageNames_Sorted = [ImageNames_Split[i][1]
for i in range(len(ImageNames_Split))]

4
 # Getting all image's name present inside the folder.
for i in range(len(ImageNames_Sorted)):
ImageName = ImageNames_Sorted[i]
# Reading images one by one.
InputImage = cv2.imread(ImageFolderPath + "/" + ImageName)

# Checking if image is read

if InputImage is None:
print("Not able to read image: {}".format(ImageName))
extt(0)

# Storing images.
Images.append(InputImage)

# If it is not folder(Invalid Path).

else:
print("\nEnter valid Image Folder Path.\n")

if len(Images) < 2:
print("\nNot enough images found. Please provide
2 or more images.\n")
extt(1)

return Images

Here, we are first reading all the image names, splitting them to the integer value of the
name and the image name, and storing them in a 2D array ImageNames_Split. Then we
are sorting the image names in increasing order and then reading the images and storing
them in a list. Thus, the images are stored such that in the list Images:
0 - “1.jpg”
1 - “2.jpg”, and so on.

We are now set with the initial code, now let’s see the changes that need to be done in part
1’s code to get the wide view panorama.

5
Cylindrical Projection and Unroll
In case of stitching only 2 images together to create a panorama, we didn’t face any difficulty
and it was a straight forward process. But in case of stitching many images together, we see
that as we stitch more and more images, images near to the side starts getting distorted as
shown.

This distortion is happening only with 4 images(2nd image is also distorted), just wonder
what will happen when we stitch 8 images for this panorama. The reason for this and its
solution’s explanation can be understood easily in this YouTube video by Joseph Redmon.
See the video from 50:26 - 1:07:23.

The method explained involves the projection of the images onto a cylinder, unrolling them,
and then stitching them together. However, depending on the input and required output, you
can even project the images onto a sphere instead of a cylinder. I have in this project only
projected the image onto the cylinder.

The projection is done using the equations given in the video itself as shown below.

6
Rearranging the equations, we get:

Using these equations, we will find the corresponding coordinates of all the points in the
transformed image(unrolled cylindrical image) in the initial input image. If these points are
outside the initial image, we will ignore them(black colour n the transformed image), else, we
will perform a bilinear interpolation and determine the color value at that coordinate in the
transformed image.

This is what is done with the code below. Note that to speed up the process, we have used
vectorization.

def Convert_xy(x, y):

global center, f

xt = ( f * np.tan( (x - center[0]) / f ) ) + center[0]

yt = ( (y - center[1]) / np.cos( (x - center[0]) / f ) ) +
center[1]

return xt, yt

def ProjectOntoCylinder(InitialImage):
global w, h, center, f
h, w = InitialImage.shape[:2]

7
center = [w // 2, h // 2]
f = 1100

# Creating a blank transformed image

TransformedImage = np.zeros(InitialImage.shape, dtype=np.uint8)

# Storing all coordinates of the transformed image in

# 2 arrays (x and y coordinates)
AllCoordinates_of_ti = np.array([np.array([i, j])
for i in range(w) for j in range(h)])
ti_x = AllCoordinates_of_ti[:, 0]
ti_y = AllCoordinates_of_ti[:, 1]

# Finding corresponding coordinates of the transformed

# image in the initial image
ii_x, ii_y = Convert_xy(ti_x, ti_y)

# Rounding off the coordinate values to get exact pixel

# values (top-left corner)
ii_tl_x = ii_x.astype(int)
ii_tl_y = ii_y.astype(int)

# Finding transformed image points whose corresponding

# initial image points lies inside the initial image
GoodIndices = (ii_tl_x >= 0) * (ii_tl_x <= (w-2)) * \
(ii_tl_y >= 0) * (ii_tl_y <= (h-2))

# Removing all the outside points from everywhere

ti_x = ti_x[GoodIndices]
ti_y = ti_y[GoodIndices]

ii_x = ii_x[GoodIndices]
ii_y = ii_y[GoodIndices]

ii_tl_x = ii_tl_x[GoodIndices]
ii_tl_y = ii_tl_y[GoodIndices]

# Bilinear interpolation
dx = ii_x - ii_tl_x
dy = ii_y - ii_tl_y

weight_tl = (1.0 - dx) * (1.0 - dy)

weight_tr = (dx) * (1.0 - dy)

8
 weight_bl = (1.0 - dx) * (dy)
weight_br = (dx) * (dy)

TransformedImage[ti_y, ti_x, :] =
(weight_tl[:, None] * InitialImage[ii_tl_y, ii_tl_x, :]) + \
(weight_tr[:, None] * InitialImage[ii_tl_y, ii_tl_x+1, :]) + \
(weight_bl[:, None] * InitialImage[ii_tl_y+1, ii_tl_x, :]) + \
(weight_br[:, None] * InitialImage[ii_tl_y+1, ii_tl_x+1, :])

The correct value of f for an image set is determined by the hit-and-trial method.

A final step that needs to be done is the cropping of the transformed image. The
TransformdImage obtained by the code above is as follows:

The black region on the left and right also needs to be eliminated. For this, I just used a trick
and exploited symmetry to my benefit.

# Getting x coorinate to remove black region from right and

# left in the transformed image
min_x = min(ti_x)

# Cropping out the black region from both sides

# (using symmetricity)
TransformedImage = TransformedImage[:, min_x : -min_x, :]

return TransformedImage, ti_x-min_x, ti_y

9
Here, min_x holds the minimum value of x coordinate of the transformed image whose
corresponding point found using the above equations lie inside the initial image. As it is
noted that the left and right boundaries are vertical and equidistant from the respective
boundaries, direct cropping of the image is done using only the minimum x coordinate value.
Thus, the image now obtained is:

Now let us use these transformed images to create a panorama.

10
Stitching the images
The main algo for stitching the images, i.e. finding matches between the images, finding the
homography, and correcting the frame size and the homography remains the same as
discussed in part 1 but the algo for overlapping the images is modified.

We earlier overlapped the images directly, that is just placed the base image over the
transformed secondary image directly. Now the previous method will not work as now the
images are not rectangular. Thus, we’ll now be using the mask of the secondary image to
stitch it over the base image.

NOTE: Secondary image will be the unrolled cylindrical projection of the initial image whose
perspective transform will be done before overlapping, and Base image is the panoramic
image already obtained with the previous images.

For overlapping of two images, the mask of any one image is enough. Thus, we will be
having the mask of the secondary image as it will be easier to create/track as compared to
the base image.

Let’s have a look at the code block.

def StitchImages(BaseImage, SecImage):

# Applying Cylindrical projection on SecImage
SecImage_Cyl, mask_x, mask_y = ProjectOntoCylinder(SecImage)

# Getting SecImage Mask

SecImage_Mask = np.zeros(SecImage_Cyl.shape, dtype=np.uint8)
SecImage_Mask[mask_y, mask_x, :] = 255

# Finding matches between the 2 images and their keypoints

Matches, BaseImage_kp, SecImage_kp = FindMatches(BaseImage,
SecImage_Cyl)

# Finding homography matrix.

HomographyMatrix, Status = FindHomography(Matches, BaseImage_kp,
SecImage_kp)

# Finding size of new frame of stitched images and updating

# the homography matrix
NewFrameSize, Correction, HomographyMatrix =
GetNewFrameSizeAndMatrix(HomographyMatrix,
SecImage_Cyl.shape[:2],
BaseImage.shape[:2])

11
 # Finally placing the images upon one another.
SecImage_Transformed = cv2.warpPerspective(SecImage_Cyl,
HomographyMatrix,
(NewFrameSize[1], NewFrameSize[0]))
SecImage_Transformed_Mask = cv2.warpPerspective(SecImage_Mask,
HomographyMatrix,
(NewFrameSize[1], NewFrameSize[0]))
BaseImage_Transformed = np.zeros((NewFrameSize[0],
NewFrameSize[1], 3),
dtype=np.uint8)
BaseImage_Transformed
[Correction[1]:Correction[1]+BaseImage.shape[0],
Correction[0]:Correction[0]+BaseImage.shape[1]]
= BaseImage

StitchedImage = cv2.bitwise_or(SecImage_Transformed,
cv2.bitwise_and(BaseImage_Transformed,
cv2.bitwise_not(SecImage_Transformed_Mask)))

return StitchedImage

Firstly, the secondary image is projected onto the cylinder and unrolled, and its mask image
is also obtained using the data points obtained from the projection. Then the matches and
the homography are found as discussed in part 1 and also the new frame size is calculated
and the homography is corrected.

Afterward, the perspective transform of the secondary image and its mask image is done
and a blank frame(BaseImage_Transformed) is also created of the same dimensions in
which the base image is placed at the correct position. The next line of code overlaps these
two images now using the mask properly. The logic is explained in the diagram below.

12
We finally obtain the Stitched Image which will again be used as the BaseImage for the next
secondary image thus creating a large wide-view panorama without distortion.

Future scopes
Is it done? No, it is not.

This project can be continued for obtaining even better results. Some of the ways are
discussed below.

● The condition of naming the images in increasing order wrt their adjacent images can
be removed.
● Projecting the image onto a sphere instead of cylinder would give even better results.
● Determination of the focal length(for projection) by hit and trial can be automated.
● You notice the lines at the boundaries if the images in the final panorama? Yes, these
lines can be removed by applying Gain Compensation and Multi-Band Blending.
These methods are discussed in the research paper Automatic Panoramic Image
Stitching using Invariant Features by Matthew Brown and David G. Lowe. The
research paper is shared in the GitHub repository of this project.
● Applying Automatic Panorama Straightening(also discssed in the research paper).

13