Panorama Stitching

Part I (Stitch 2 Images)

Image Processing - OpenCV, Python & C++

By: Rahul Kedia

Source Code:
https://github.com/KEDIARAHUL135/PanoramaStitching
.git

1
Table of Contents

Overview 3

Problem Statement 4

Approach 4

Setting up initial code 5

Finding matches between the two images 6

Finding Homography 8

Understanding the problem 10

Re-calculating Homography and Stitched frame size 12

Stitching the images 15

2
Overview

Images have been a part of our lives for a long time now and we cannot imagine a time we
got out with our friends and family and not having clicked a photo with them or not taking a
picture of the beautiful scenery. Having a huge group of friends fit in a photo clearly or
capturing a photo of the scenery is sometimes difficult. Panoramic images come in handy in
such cases.

Panoramic images are basically the wide-angle view of the scene. These are nowadays
possible using the Panorama feature inbuilt in mobile cameras. But what if you have different
images of the scene and you want them together as a panorama. This is possible using
different image stitching techniques available, one of which is discussed in this project where
we will use the OpenCV library and stitch 2 images together side by side to get a panoramic
image as shown.

Multiple images can also be stitched which will be discussed in the second part of this
project and we also will overcome some drawbacks of this technique in the second part. Let
us for now have a look at the simpler concept of how to stitch 2 images together first. We’ll
use this knowledge and code to build up code for multiple image stitching.

3
Problem Statement
Given two images of the same scene with some parts of the images overlapping with each
other, we will be stitching these two images side-by-side to create a panoramic image.

Note that this code will stitch only two images. This will make you familiar with the concept
and approach. After this, in the second part of the project, we will see how can we stitch
multiple images to create a single large panorama.

Approach
To solve this problem, we will implement the following steps:

● Firstly, after reading the two images, we will find good matches between the two
images using SIFT features -> Brute Force Matcher -> D.Lowe’s Ratio Test.
● Then we will find the homography matrix of the secondary image using which it will
be transformed and stitched upon the base image.
NOTE: Base Image is the image over which the other image(secondary image) will
be overlapped.
● The Homography matrix found in the previous step overlaps the image but it crops
the image going out of the frame and also at this time we don’t know the new frame
size. Thus, in the final step, we will find the new frame’s (stitched frame’s) size,
correct the homography matrix, and finally stitch the images.

4
Setting up initial code
First of all, we will have to make a basic code that imports all the required libraries and read
the two input images, and then pass the images for processing. In this section, we will see
how to do it.

First of all, create a new directory for this project. I am naming it “PanoramaStitching” and
then create a file in it that will contain our source code. I am naming this file as “main.py”.
Also, add a folder in this directory containing all of our input images. I have added the input
images in the folder named “InputImages”. In this folder, the images of the same scenes
are further separated into different folders.

Now in the main.py file, let’s write the code that will read the two input images and pass
them for stitching.

Let us first start by importing the libraries required.

import cv2
import numpy as np
from matplotlib import pyplot as plt

Here I have imported the cv2, numpy, and matplotlib.pyplot libraries. The cv2 and numpy
libraries will be used throughout the project for stitching and the matplotlib.pyplot library will
be used to finally view the stitched images.

Now let’s build up the code that will read the input images and pass them for further
processing. The code is explained by the comments.

if __name__ == "__main__":
# Reading the 2 images.
Image1 = cv2.imread("InputImages/Sun/1.jpg")
Image2 = cv2.imread("InputImages/Sun/2.jpg")

# Checking if images are read properly.

if Image1 is None or Image2 is None:
print("\nImages not read properly or does not exist.\n")
exit(0)

# Calling function for stitching images.

StitchedImage = StitchImages(Image1, Image2)

Now with the help of the above code, we are ready to build up the code for stitching the two
images. Let’s see how it is done.

5
Finding matches between the two images
The two images are passed to a function FindMatches that is responsible for finding the
good matches between the two images using SIFT features -> Brute Force Matcher ->
D.Lowe’s Ratio Test.

(Here, Image1 -> BaseImage; Image2 -> SecImage)

Let us first find the keypoints and descriptors of the images individually using the SIFT
algorithm.

# Using SIFT to find the keypoints and descriptors in the images

Sift = cv2.SIFT_create()
# “kp” is for keypoints and “des” is for descriptors.
# NOTE: We are first converting the coloured BGR image to grayscale
# and then passing them in Sift.detectAndCompute() function.
BaseImage_kp, BaseImage_des = Sift.detectAndCompute(cv2.cvtColor(BaseImage,
cv2.COLOR_BGR2GRAY),
None)
SecImage_kp, SecImage_des = Sift.detectAndCompute(cv2.cvtColor(SecImage,
cv2.COLOR_BGR2GRAY),
None)

The Scale-Invariant Feature Transform (SIFT) is a feature detection algorithm in computer

vision to detect and describe local features in images. To know more about the SIFT
algorithm, you can refer to the Wikipedia article and the OpenCV documentation on SIFT.

After finding the keypoints and descriptors for the two images, let us compare them and find
matches between them using the Brute-Force Matcher technique. Matching basically
refers to finding the same (common) keypoint (same point in the images) in both images.

Brute-Force Matcher takes the descriptor of one feature in the first set and is matched with
all other features in the second set using some distance calculation, and the closest one is
returned. Refer to the OpenCV documentation on Brute-Force Matcher to know more in
detail.

# Using Brute Force matcher to find matches.

BF_Matcher = cv2.BFMatcher()
InitialMatches = BF_Matcher.knnMatch(BaseImage_des, SecImage_des, k=2)

6
Here, “InitialMatches” store all the matches found by the Brute-Force Matcher. Now we will
have to filter out the good matches only from these so that the final transformation is not
affected by incorrect matches found.

This filtering of matches will be done by using the Ratio Test by D.Lowe. Let’s have a look
at the code.

# Applying ratio test and filtering out the good matches.

GoodMatches = []
for m, n in InitialMatches:
if m.distance < 0.75 * n.distance:
GoodMatches.append([m])

Here, “GoodMatches” will contain the matches whose first best match is even smaller than
0.75 times of the second-best match for a keypoint. To know more about the working of this
test, have a look at this simple but wonderful Stackoverflow answer.

Now let’s move ahead to our second step where we will find the Homography for the
transformation of the secondary image.

7
Finding Homography
The matches found in the previous step will now be used to find the homography matrix
used to transform the secondary image to overlap the base image in order to give a
panoramic view thus, the stitched images.

Before finding the homography matrix, first, let us extract the coordinates of matches (initial
and final coordinates) in a correct format. In simpler words, these initial and final coordinates
will be used to find a transformation(homography) matrix that will later be used to transform
the secondary image onto the base image.

# If less than 4 matches found, exit the code.

if len(Matches) < 4:
print("\nNot enough matches found between the images.\n")
exit(0)

# Storing coordinates of points corresponding to the matches found

# in both the images
BaseImage_pts = []
SecImage_pts = []
for Match in Matches:
BaseImage_pts.append(BaseImage_kp[Match[0].queryIdx].pt)
SecImage_pts.append(SecImage_kp[Match[0].trainIdx].pt)

# Changing the datatype to "float32" for finding homography

BaseImage_pts = np.float32(BaseImage_pts)
SecImage_pts = np.float32(SecImage_pts)

First, the number of matches found is checked. There must be atleast 4 matches in order to
find a correct homography matrix. Then the initial and final coordinates of the points are
extracted from the base image and secondary image keypoints found earlier and these
points are converted to float32 datatype for further calculation.

Now let us find the homography matrix using the OpenCV function findHomography().

# Finding the homography matrix(transformation matrix).

(HomographyMatrix, Status) = cv2.findHomography(SecImage_pts, BaseImage_pts,
cv2.RANSAC, 4.0)

8
It is to note that there still can be matches that are not linear with the rest of the matches and
if used for calculation of homography matrix, they may pose some error. To resolve this
issue, the findHomography() function requests a flag (RANSAC in our case) and this will first
filter out the correct matches(inliers) and then find the homography matrix using them. To
know more about this function, you can refer to the official OpenCV documentation for
findHomography() function.

Now let’s see how the image is overlapped using this homography matrix. Here we have
simply increased the width of the base image by the width of the secondary image for
accommodating the secondary image.

As you can notice, first of all, there is a lot of unnecessary black space on the right and we
as of now cannot estimate the correct width of the stitched image as of now. Also, as
compared to the output image shown in the beginning, this image has lost a lot of data of the
secondary image. The cloud on the top right corner is cut in half and the ground below is
also lost a little bit. This problem doesn’t seem to be big in our case but what if in actual
scenarios, we lose some important data?

Here, we will have to correct the stitched frame size, and also we will have to re-calculate
the homography matrix to accommodate the secondary image completely inside the frame.
Let us see how we can do this in the next parts.

9
Understanding the problem

Before running into the code, let us first understand where this problem is arising and what
we are doing to solve this problem.

Below are the two initial separate images we are taking. The height, width, and coordinates
of the four corners of the secondary image are shown.

Now using the homography matrix, the secondary image will be transformed and overlapped
over the base image so you can think of it as the 4 corners of the secondary image will be
projected in such a way that the image will be overlapped.

Let us see what will be the final coordinates of these 4 corners of the secondary image using
the homography matrix found for the combination of these 2 images.

10
In the above image, the red frame is the resultant image obtained from the code until the
previous part. The blue frame is the required final panoramic frame. The green coordinate
points are the points obtained after transforming the secondary image using the homography
matrix obtained earlier.

As you can see, the top right corner of the transformed secondary image has a negative
y-coordinate showing that the image is going out of the current frame (red frame). Also, the
bottom right corner has a huge y-coordinate value due to which it is going out of the current
frame.

Our challenge is to calculate a homography matrix that transforms the secondary image to a
similar position wrt the base image and also we have to keep the total stitched image
confined inside the stitched frame for which we will require the stitched frame size
beforehand.

To solve this problem, we will first calculate and find the position of the corners using the
homography matrix, shift these coordinates such that each point has positive coordinates.
Using these shifted positive coordinates, we will find the new homography matrix and the
dimensions of the new frame (blue frame).

Let’s have a look at the code for this in the next part.

11
Re-calculating Homography and Stitched frame
size

We will first find the coordinates of the corners of the secondary image after transformation
is done using the current homography matrix. The initial coordinates “src(x, y)” are
transformed to the final coordinates “dst(x, y)” using the homography matrix (M(3x3)) using
the formula given below.

[Source: OpenCV documentation]

# Reading the size of the image

(Height, Width) = Sec_ImageShape

# Taking the matrix of initial coordinates of the corners of the

# secondary image
# Stored in the following format:
# [[x1, x2, x3, x4], [y1, y2, y3, y4], [1, 1, 1, 1]]
# Where (xi, yi) is the coordinate of the i th corner of the image.
InitialMatrix = np.array([[0, Width - 1, Width - 1, 0],
[0, 0, Height - 1, Height - 1],
[1, 1, 1, 1]])

# Finding the final coordinates (xi, yi) of the corners of the

# image after transformation.
FinalMatrix = np.dot(HomographyMatrix, InitialMatrix)

[x, y, c] = FinalMatrix
x = np.divide(x, c)
y = np.divide(y, c)

Now let us find the dimensions of the new stitched frame and the correction factor. The
correction factor holds the position of the top left corner of the base image in the final
stitched frame.

12
# Finding the dimentions of the stitched image frame and the
# "Correction" factor
min_x, max_x = int(round(min(x))), int(round(max(x)))
min_y, max_y = int(round(min(y))), int(round(max(y)))

New_Width = max_x
New_Height = max_y
Correction = [0, 0]
if min_x < 0:
New_Width -= min_x
Correction[0] = abs(min_x)
if min_y < 0:
New_Height -= min_y
Correction[1] = abs(min_y)

# Again correcting New_Width and New_Height

# Helpful when secondary image is overlaped on the left hand side
# of the Base image.
if New_Width < Base_ImageShape[1] + Correction[0]:
New_Width = Base_ImageShape[1] + Correction[0]
if New_Height < Base_ImageShape[0] + Correction[1]:
New_Height = Base_ImageShape[0] + Correction[1]

The logic applied here is simple and self-explanatory. Draw this problem out on a rough
paper and try to understand the logic.

Let us now find the coordinates of the 4 corners of the secondary image the should be
obtained after the transformation in the new frame.

# Finding the coordinates of the corners of the image if they all

# were within the frame.
x = np.add(x, Correction[0])
y = np.add(y, Correction[1])
OldInitialPoints = np.float32([[0, 0],
[Width - 1, 0],
[Width - 1, Height - 1],
[0, Height - 1]])
NewFinalPonts = np.float32(np.array([x, y]).transpose())

13
Finally, using these new coordinates, let us obtain the corrected/new homography matrix.

HomographyMatrix = cv2.getPerspectiveTransform(OldInitialPoints,
NewFinalPonts)

It is to note that earlier we used the function findHomography() to find the homography
matrix whereas now we are using the function getPerspectiveTransform(). This is because
findHomography() can filter out the correct points (can remove outlier matches) if more than
4 points are provided whereas getPerspectiveTransform() will find the homography matrix
directly using the 4 points provided considering that they are correct as per our need.

Let us now obtain the final stitched panoramic image using the new homography matrix.

14
Stitching the images

Let us now have a look at the code that finally stitches the two images using the values
obtained previously.

# Finally placing the images upon one another.

StitchedImage = cv2.warpPerspective(SecImage, HomographyMatrix,
(NewFrameSize[1], NewFrameSize[0]))
StitchedImage[Correction[1]:Correction[1]+BaseImage.shape[0],
Correction[0]:Correction[0]+BaseImage.shape[1]] = BaseImage

Below is the final result obtained.

The two images are stitched successfully into a panoramic view.

It is to note that this code stitches only 2 images and it is giving somewhat a 3D cylindrical
look. We will take care of this problem and modify the code to stitch as many images as
possible together at once in the second part of this project.

15