Spherical Image Stitching Based on ORB and PROSAC
Algorithm
Li Xin Liu Yarong
Guangxi key Laboratory of EmbeddedGuangxi key Laboratory of Embedded
Technology and Intelligent System Technology and Intelligent System
Guilin University of Technology Guilin University of Technology
China 541004 China 541004
1996019@glut.edu.cn 2002048@glut.edu.cn
Yang Liyan Yang Xiaofei
Guangxi key Laboratory of EmbeddedGuangxi key Laboratory of Embedded
Technology and Intelligent System Technology and Intelligent System
Guilin University of Technology Guilin University of Technology
China 541004 China 541004
2014038@glut.edu.cn 2006016@glut.edu.cn
ABSTRACT
This paper provides a real-time image stitching method. During
stitching users often discover that panoramic picture is not ideal
because of the spherical camera taking the picture of wide-angle
and large-area distortion. We presents the ORB algorithm
extracting feature points, and PROSAC algorithm purification.
While traditional stitching researchers use the SIFT or SURF
method detecting feature points, and the RANSAC algorithm
removes outpoints. We discover that SIFT and SURF algorithms
require very complex calculations, and the RANSAC algorithm is
equally and random for all feature point pairs, so the real-time
processing is slow. In this paper, the use of ORB and non-random
PROSAC algorithm greatly shortens the running time. The
SURF+RANSAC and ORB+PROSAC algorithms are used to
compare the stitching speed. Experiments show that the
ORB+PROSAC algorithm increase matching speed with better
matching accuracy.
CCS Concepts
• Computing methodologies ➝ Computer graphics ➝
Image manipulation➝Image Processing
Keywords
image stitching; spherical image correction; ORB; PROSAC;
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from
Permissions@acm.org.
ICIIP '18, May 19–20, 2018, Guilin, China
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-6496-6/18/05…$15.00
https://doi.org/10.1145/3232116.3232142
160
Li Donghui
Guilin University of Technology
China 541004
914490624@qq.com
Wang Yuchen
Guilin University of Technology
China 541004
2846140310@qq.com
1. INTRODUCTION
To allow spherical camera users to take pictures with a viewing
angle wider than the one supported by the mobile lens, researchers
have proposed techniques for stitching multiple photographs or a
stream of video into a single contiguous panoramic picture, or
panorama [1].
The traditional image mosaic use the SIFT and SURF detection
feature points, combined with RANSAC algorithm excluding
points to increase the accuracy of matching feature point set[2].
However, these conventional methods require a lot of complex
calculations, for higher real-time systems, these methods have a
certain degree of difficulty.
To solve the above problems, we proposes a method of using the
ORB and PROSAC panoramic image stitching algorithm which
increasing the detecting and removing outer point speed of the
feature point, shortening the time to the entire splicing process,
and using the fewer resources to achieve real-time process.
2. Algorithm
2.1 Longitude mapping method for spherical
image correction
Spherical coordinate positioning method, also called longitude
mapping method, is a typical fast two-dimensional fisheye image
correction algorithm.
As shown in Fig1, using the principle of spherical longitude
mapping[1], there should be these following steps :

Figure 1. image correction longitude module.
2.1.1 Model establishment of spherical coordinate
positioning algorithm;
Each line in the graph is a common feature in the same meridian
point, after the correction, the correction of coordinates, with the
abscissa, the same as H, K of the two, although in the horizontal
and vertical coordinates of different, but in fact they are corrected
point horizontal coordinates are the same. The main principle of
the model is: in the picture, Y take any vertical direction, or both
from the right and left spherical, the angle difference is consistent,
and the corresponding dx segment on the x axis uniform division
of warp, make different y on the x direction of the distance
between the longitude is equal.
2.1.2 Binaryzation of image and contour extraction;
The binaryzation of the image is the process of turning a color
RGB image into a pure gray image. The steps of this algorithm
are: first step is setting the threshold and establish the
corresponding scan loop. For the image of RGB system, gray
scale transformation is needed, and the transformation formula is
as follows:
I=0.59R+0.11G+0.3B (1)
According to formula (1), the gray values of each point can be
obtained, and then the gray value difference I can be obtained.
Once the value of I is found to be greater than the previously set
threshold, the scan is considered to have entered the active region,
and the search in that direction will cease immediately.
At the end of the scan, the circumscribed square is obtained, and
the center and radius formulas are calculated as follows:
xI  xr 
x0 
2 

y  yb 
y0  t (2)
2 

x  xI 
Rx  r
2 
yb  y1 
Ry   must, in order to obtain better correction effect of bilinear
2 
2.1.3 Solution of spherical coordinate positioning
algorithm;
As shown in Fig. 1, according to the line segment represented in
the diagram and the principle described in the preceding section,
the following formula shall be applied:
161
xk xh 
 
dx R (3)

xh 2 2 
xk   R  Y
R 
Among them, R is the radius of the effective area of the spherical
camera, h is the abscissa of the original image (as an integer), and
Y is the ordinate of the k-point in the original image. Convert to
inverse mapping like xh:
xk  R
xh 
R2  Y 2 (4)
The inverse mapping finds the non-integer pixels in the original
image. Then, through the corresponding interpolation method, one
or more accurate points in the original image can be located.
Through the values of these points, the pixels after the correction
are obtained.
2.1.4 Description of spherical coordinate positioning
algorithm
(1) Obtaining the image and process the two values. In this paper,
the classical two-dimensional spherical coordinate positioning
method is used to deal with the spherical image, without the
camera calibration and other related operations;
(2) The radius and center of the effective region are obtained by
using the scan line approximation method. After the two valued
image is scanned by the scanning line method, it can distinguish
the effective area more clearly, and overcome the influence of the
edge black spot, and realize the similar effect with the region
growth method. If the valid area is detected as an ellipse, then the
standard circle transformation of the ellipse is needed;
(3) After obtaining the radius and the center, the mapping relation
from the target image to the original image is established. First, a
square target image is determined by the radius, and the initial
pixel is set at 0.;
(4) In pixels, coordinates are established in the target image and
the original image, and inverse mapping is achieved according to
formula (4);
(5) Getting the relationship between the target image and the
original image of the corresponding pixel;
(6) Obtaining the initial image correction.
2.1.5 Correction of image processing
The model and algorithm of spherical coordinate orientation of
spherical image distortion correction, image restoration of middle
part of complete, but there are still some edge barrel distortion
interpolation of the image gray level after correction.
2.1.6 experimental result
In this paper, a spherical surface projection correction method and
a spherical coordinate positioning method are used to correct the
distortion of the same spherical image. The size of the image is
300 * 400 pixel. The results are shown in figures 3 and 4. Figure 2
shows the distorted spherical image artwork. Fig 3 and Fig 4 show
the effect of distortion correction by spherical perspective
projection and spherical coordinate positioning.

Figure 2. spherical image distortion
Figure 3. Spherical perspective projection correcting image
Figure 4. Longitude correcting image
You can see from Figure 4, the middle region of the image
correction of spherical coordinate positioning method is better, the
bilinear interpolation of the image gray scale correction after the
bending distortion curve edge also got a good correction. As
shown in Figure 3, the spherical perspective projection method is
not clear enough, and the loss of edge information is relatively
serious, while the spherical coordinate positioning method is more
detailed for the restoration of distorted scenes. Therefore, it can be
concluded that the spherical coordinate positioning model is better
than the spherical perspective projection model.
2.2 Improving ORB algorithm
2.2.1 Feature point detection
The first step is feature detection based on ORB. In the traditional
image stitching technology, SIFT and SURF are mainly used for
feature point detection [2, 3, 4], and the RANSAC algorithm is
used to remove the exclusion points to increase the accuracy of
162
matching feature point sets. However, these traditional methods
require a large number of complex operations. For systems with
high real-time requirements, these methods have certain
difficulties. ORB algorithm is combined by o-FAST Corner
Detection and BRIEF algorithm[5]. The algorithm uses Harris
corner detect feature. FAST defined as: If a pixel compare to its
surrounding neighborhood have a sufficient number of pixels in
different regions, the point may be corner.
As Figure 5, select a pixel P from the image, set the value X(P).
Compare this pixel to 16 pixles surrounding P. P as the circle
center, there are N consecutive pixels on the circle, the pixel value
either biggest than X(P)+T(T is threshold) or smallest than X(P)-T,
then P is a corner. N is 9 or 12.
Figure 5. FAST Corner detection .
o-FAST uses Rosin gray centroid, in which an image area defined
in a moment, where I (x, y) is value of point (x, y)：
㌳䁠
䁠 ㌳ ㌳䁠晦 (5)
Obtained the centroid of image:
C=(Cx,Cy), ， 䁠 (6)
In ORB, above image region is the neighborhood of FAST key
point, that is the key point of a circle, with three circular area
radius pixels. x, y in the range [3,3], that is, within the circular
area. This get a vector OC, OC represente direction of the FAST
key points, value of atan2 is the arctangent function range in (-n,
n), θ is key direction of the FAST.
θ=arctan（m01,m10) (7)
2.2.2 the feature descriptors of ORB
ORB use improved BRIEF to description the feature points,
rBRIEF. The algorithm use image smoothing before feature
described. The principle is to find a good match point from all
corners detected by o-FAST algorithm to describe the
characteristics of the image. ORB algorithm use the 256bits
descriptor, in order to obtain information BRIEF direction, you
first need to set the dot matrix form. For an n-bit set of test points
(xi,yi)，defines a matrix of 2 × n：
 x1 , x2 , xn 
S    (8)
 y1 , y2 , yn 
Depending on the direction angle obtained from FAST θ ， find
out its corresponding rotation matrix R θ ， this get the set of the
test which with the directional characteristics Sθ=RθS. ORB called
steered BRIEF, new descriptor can be expressed as：
 g n ( p, )  f n ( p) | ( xi , yi )  S (9)
Among them, p (x) is the average gray value of the image block p
at point (xi,yi). When the N test point is selected, finally the
descriptor includes the main direction theta, the main direction
information to the descriptor.
2.2.3 feature point matching.
After calculating the descriptors, feature points can be matched
by comparing the similarity of descriptors. The Hamming distance
is used in the ORB to represent the similarity of descriptors. In
this experiment, the feature point rough matching is performed by
using the ratio T between the nearest neighbor distance and the
second nearest neighbor distance. When T is less than the
threshold w (usually between 0.6 and 0.8), the feature point
corresponding to the nearest neighbor distance is considered to be
a rough matching point pair.
(a) ORB rough matching .
(1) Each pair of data in the matching point set is sorted according
to the distance from the new point to the corresponding matching
point after the matrix transformation;
(2) According to the number of repeated sampling t, the size of
the generation set n that matches the correct model point is
determined, and a sample set is formed by extracting 7 matching
points and the nth matching point from the top n of the ordered
matching point set;
(3) Calculate the model parameters according to the sample set,
calculate the distance from the new point to the corresponding
matching point after the remaining matching points have been
transformed by the model, and count the matching point logarithm
X within the allowable error range;
(4) Repeat the above steps. When the ratio of the interior point
reaches the maximum and is greater than the preset threshold T or
the number of interior points no longer increases, the
corresponding matching point pair set is the maximum interior
point set, and the final transformation matrix is calculated in the
interior point set.
To verify actual effect of this algorithm, test the purified
experiment with above two images, and count the operation
results. Precise matching effect as shown in Figure.5:
To count the purified time, using getTickCount( ),
getTickFrequency( ) to get times:
double t = getTickCount();
//start count time
… …;
// Required count time part

(b) SURF rough matching .

t=((double)getTickCount()-t)/

Figure 6. ORB and SURF Feature point matching . getTickFrequency();

The experiment use 960x720 images to ORB rough matching, //end count time cout<<"time："<<t<<"s"<<endl;
results shown in Figure 6. In Table 1, we compare their times of
per thousand features, exist obviously mismatch match, we need //output the PROSAC running time
PROSAC algorithm to remove the mismatch features. In the experiment, the ORB+PROSAC and SURF+PROSAC
Table 1. ORB and SURF comparing algorithms were used to match the two original images in Figure 6,
and their matching speeds were compared. The results are shown
Thousand in the figure 7.
Match
Algorithm features Run time points
points
runtime
ORB 500 161 345.2ms 690.6ms

SURF 796 398 1726.48ms 2168ms

2.2.4 Using PROSAC algorithm to improve (a) PROSAC fine matching .

matching accuracy
The idea of PROSAC algorithm [6] is to first sort the correlation
between sampling points and models, then iterate according to the
sampling points with large correlation, and finally calculate the
model parameters based on the data of the interior points. The
number of interior points represents the model parameters, good
or bad. After repeated verification many times, until the number
of internal points no longer increases, the matching result is
optimal at this time. (b) RANSAC fine matching .
The algorithm is implemented as follows: Figure 7. 960x720 pixels image matching .

163
Comparing the above results of rough matching between ORB
and SURF (Table 1), it can be clearly seen that some mismatching
feature points are removed after several iterations of fine-match
iteration, and both algorithms have achieved very good removal
effects ( As shown in Table 2 and Table 3), but the PROSAC
algorithm has a significant speed advantage over the RANSAC
algorithm.
Table 2. ORB+PROSAC matching
PROSAC feature Inner
Total time
time points points
350.668ms 5.327ms 100 80
Table 3. SURF+RANSAC matching
PROSAC feature Inner
Total time
time points points
1774.85ms 48.365ms 100 92
The RANSAC treats all sampling points equality, but PROSAC
sort model orrelation and sampling points first. Then iterate the
big correlation points, final calculate model parameters by inner
point, more inner points represents high quality of the model
parameters. After replicated several times, until no increase in
inner point, then get a optimal model parameters. PROSAC
assumes correct sampling points is more than error points [10],
that are inner points more than outer points. The ORB itself, the
inner points higher than the outer points, therefore PROSAC is
very suitable.
2.3 Image fusion
Image fusion is to merge the registered images according to the
alignment position, and eliminate the color discontinuity and the
influence of noise on the image. Commonly used fusion methods :
Multi-resolution spline method [7]; Weighted average method [8];
Poisson fusion method [9]. The multi-resolution spline method is
a color fusion method because it involves the structural problems
of Costa and Laplace, which has the disadvantages of large
amount of calculation and time-consuming. The weighted average
method is relatively simple, but it can easily cause obvious
stitching seams. Poisson fusion is simple and convenient when
selecting fusion regions. Fusion uses the gradient field in the
source image block as a guide, and the difference between the
source image and the target image on the fusion boundary is
smoothly diffused into the fused image block, making the fused
image block. Achieve a seamless effect when blended into the
target image, and its hue and lighting are consistent with the target
image. Therefore, in this paper, the spliced image is smoothed by
the progressive weighted average algorithm.
The camera product is Hikvision No.DS-2CD3942F-I. The fusion
images captured with two overlapping areas are shown in the
following figure 8.
164
Figure 8. panoramic picture with two spherical images.
The experimental results show that by the processing of the
corrected fish-eye image, there is no obvious transition stitching at
the overlapping area, the brightness of both sides is basically the
same, thus achieving a smooth transition of the image pixels, the
experiment has been achieved better splicing effect. Since the
right image is transformed using the transformation matrix H and
then stitched with the left image, the leftmost image of the spliced
image results. A black blank area appears on the side.
3. Conclusion
In this paper, ORB feature extraction and PROSAC algorithm are
studied. ORB combines the oFAST operator and rBRIEF
descriptor to achieve rough matching of the image. The PROSAC
algorithm achieves the purification of matching feature points and
uses 960x720 pictures for experiments on ORB+PROSAC and
The SURF+RANSAC algorithm is compared to calculate the total
time of fine matching under 100 feature points, the time of
purification algorithm and the number of internal points to
evaluate the performance of the algorithm.
According to the description in [10], the speed of ORB algorithm
is 10 times that of SURF, and PROSAC is nearly an order of
magnitude faster than RANSAC algorithm speed. Then ORB
combined with PROSAC algorithm can achieve faster image
stitching than SURF and RANSAC algorithm. Therefore, this
algorithm can achieve good real-time performance and can
guarantee the fluency of real-time splicing. It can be used for real-
time splicing of images and videos.
4. ACKNOWLEDGMENTS
This research work was supported by Guangxi key Laboratory
Fund of Embedded Technology and Intelligent System （ Guilin
University of Technology）under Grant No.2017-2-5.
5. REFERENCES
[1] Xu Jincai, Wu Xukang. Spherical image correction method
based on spherical coordinate positioning algorithm [J].
Computer disc software and application, 2014,17 (11): 129-
130. [2017-08-19].
[2] Rublee E ， Rabaud V,Strecha C ， et al. ORB:an efficient
alternative to SIFT or SURF[C]. Computer Vision(ICCV)，
2011 IEEE International Conference on.IEEE， 2011:2564-
2571.
[3] ZHANG Yajuan. Research on Image and Video Mosaic
Technology Based on SURF Feature[D]. Xidian University,
2013.
[4] Li Qian, Jiang Zetao. Binary SIFT feature descriptors and
image mosaic optimization [J]. Journal of Image and
Graphics, 2016, 21(12): 1593-1601
[5] Edward Rosten,Reidorter,TomDrummond. Faster and Better:
A Machine Learning App -roach to Corner Detection , 2010
IEEE Transactions on pattern analysis and machine
intelligence.
[6] TONG Ying, ZHANG Yan. Image Matching Algorithm
Based on Dual Detection Strategy Coupled with PROSAC
Technology[J]. Computer Engineering and Design, 2017,
38(11): 3137-3142..
[7] Wang Liangliang, Zhang Guicang, Jia Wenxiao. Medical
Image Fusion Method Based on Gradient Fusion Rule [J/OL].
 Journal of Computer Applications, 2018(06):1-6[2018-01-
05].
[8] Ying Tong, Jin Chen. Compressive sensing image fusion in
heterogeneous sensor networks based on shearlet and
wavelet transform. EURASIP Journal on Wireless
Communications and Networking, 2017, Vol.2017 (1), pp.1-
5.
[9] Hu Gang, Zheng Yuyue, Qin Xinqiang. Image Fusion
Method Combining Local Neighborhood Properties with C-
165
BEMD[J]. Journal of Computer-Aided Design & Computer
Graphics, 2017, 29(04): 607-615.
[10] Jiang Pengfei. Video Image Mosaicing Based on ORB
Feature[D]. University of Electronic Science and
Technology, 2016.
.