Ghosting Elimination with A* Seam

Optimization in Image Stitching

Jyun-Hong Chen and Cheng-Ming Huang

Department of Electrical Engineering
National Taipei University of Technology
Taipei, Taiwan, ROC.
e-mail: cmhuang@ntut.edu.tw

Abstract—This paper presents an algorithm for eliminating the
blurred image regions during image stitching. Since the camera is
close to the objects when capturing the images in the indoor
environments, stitching the images in the indoor scenes easily
generates the blurred ghosting region. Here the reasons of
blurred region generations can be categorized into two kinds.
First, the slight camera vibration may result in the whole images
unclear over a short span of frames. These unclear frames can be
filtered out by distinguishing the clearness of the edge images
between different scales. The other reason for originating the
blurred region is that several image frames contain the same
scene from different views. There may exist the inconsistent
structures and observations in these different views. The color
blending is performed to let the seam between aligned images
smooth, however, it also makes these inconsistent region blurred,
which is called the ghosting effect. Here, an optimal seam
selection algorithm is proposed by utilizing the A* search
algorithm to find a segment path away from the inconsistent
overlap region and then remove the ghosting effect. The
experimental results validate that the proposed algorithm can
obtain the mosaic image with good quality and clear resolution.
Keywords- image stitching, ghosting elimination,
optimization.
I. INTRODUCTION
Recently, image stitching techniques, which group several
images into a mosaic image with wide field of view, have been
commonly applied on the digital cameras or mobile phones. In
general, the images for stitched together must be captured by
moving the camera smoothly, and the camera platform has to
be far away from the scene. Otherwise, the objects' appearance
would not be smooth and the scene may be discontinuous in the
mosaic image. This paper presents the algorithm to align the
images even when the camera moving with vibration and close
to the captured scene.
The aerial image usually utilizes the camera on the
underneath of the airplane to capture the scenes on the ground.
Based on the data obtained from the inertial measurement unit
(IMU), the homography matrices between images can be
evaluated for aligning these images. Since the altitude of the
airplane is high, the scene is usually assumed to be planar [11].
When the airplane is flying with low altitude, the objects are
close to the camera and cannot still be assumed as planar ones.
The motion parallax [1], which is also called the ghosting
This work was supported in part by the National Science Council, Taiwan
under the grants: NSC 100-2218-E-027-010-.
effect, is then easily introduced due to the movement and the
3D appearance of the object.
There are several researches present the solutions to
eliminate the ghosting effect in image stitching. When there is
a moving object in a static scene, the images from different
views would be aligned according to the scene. In order to
remove all but one observation of this moving object in the
mosaic image, a selective algorithm [2] is proposed to reserve
only one appearance of the object in these image patches. Some
researchers [3, 4] employ the texture analysis or the feature
matching to result in the seamless stitching, however, the color
of the mosaic image may not be consistent.
Since the ghosting effect usually appears in the overlap
region of the aligned images, finding the optimal seam [7, 8] in
the overlap region, which decides the boundary for reserving or
neglecting the content in aligned images, is an efficient method
to resolve the ghosting phenomenon. After graph cut of the
redundant region in the source image patches, the blending
process would be performed around the optimal seam to
smooth the color variance between aligned images. The edge
seam and texture of the objects would also be blurred after the image
blending. Hence, a path with minimum color variance or least
edge intensity [7, 8] is usually assigned as the optimal seam.
Considering the variance of color and edge intensity to find the
optimal seam is inefficient in the situation that there are several
ghosting regions in the image. Kwatra et. al. [9] decide the
optimal seam by avoiding the obvious objects in the
combination of Lab color image and edge image. However, this
algorithm cannot deal with the case that the colors the object
and background scene are very similar. Here, the A* algorithm
with the difference map of color and edge in the overlap region
is applied to select the optimal seam. The image difference in
the overlap region is regarded as the obstacle during planning
the seams. The selected seams could efficiently avoid the
difference in the overlap region between source images and
then remove the motion parallax.
The flow chart of the image stitching system is shown in
Fig. 1. First, from grabbed image sequence, the blurred image
frames will be filtered out. Here, the image sequence are
grabbed with 30 frames per second (FPS.), and several frames
will contain the same scene. The SIFT features in each image
are extracted to determine the relation with other frames and
estimate the homography matrix between the related pair of

978-1-4673-2588-2/12/$31.00 ©2012 IEEE 214

Authorized licensed use limited to: UNIVERSITY OF SOUTHAMPTON. Downloaded on August 17,2020 at 19:53:20 UTC from IEEE Xplore. Restrictions apply.
images. The image frames are then transformed and aligned in
the same coordinate. Since stitching a lot of images about the
same scene from different views easily results in the motion
parallax, which will be corresponding to the blurred region
after image blending. In order to eliminate the blurred ghosting
region, an optimal seam in the overlap region between each
related pair of images will be selected. In this overlap region,
the inconsistent edges between two aligned frames are
extracted as the obstacles in a 2D map. The desired seam path,
which should avoid the inconsistent edges, is planned by the
A* search algorithm. At last, the blending process is performed
along the selected seam to continuously and smoothly joint the
frames as a mosaic one.
Fig. 1 System flow chart.
The rest of this paper is organized as follow. Section 2 first
introduces the pre-process to filter out the blurred images by
analyzing the amount of edge pixels at different image scales
over time. The seam optimization, which finds a seam path and
avoids approaching the textured region by the A* search
algorithm, is described in section 3, Section 4 demonstrates the
experimental results and provides some comparisons to show
the efficiency of our de-ghosting method. The conclusion and
future work are given in section 5.
II. BLURRED IMAGE REMOVEMENT
Clear image Blurred image
Fig. 2. Comaprison between the blurred image and clear one.
The first row presents the original image, the second row is the corresponding
edge image, and the third row is the edge image with scale up two times.
215
Authorized licensed use limited to: UNIVERSITY OF SOUTHAMPTON. Downloaded on August 17,2020 at 19:53:20 UTC from IEEE Xplore. Restrictions apply.
The captured images would be unclear during the
movement and vibration of camera platform. In order to result
in a mosaic image without ghosting region, first, we should
remove the source images that are already blurred on them.
From the images about the same scene, such as Fig. 2, we can
see that the clear ones have distinguishable edges, however, the
blurred ones with weak edge response. Hence, the unclear
image can be determined by evaluating the variance of the edge
pixels from an image sequence. Moreover, when the clear
image is scale up, the quality of this image would be fine and
the details are still clear. But the details of the blurred image
would still be unclear after the image scaled up. Every image
frame will be scaled up by four times. The non-edge ratio of
the non-edge pixels over image size is evaluated, and the
variation of the non-edge ratio of each frame between different
scales is shown as Fig. 3(a).
The peaks of the curve in Fig. 3(a) will be corresponding to
the blurred image frames. In order to extract these peaks, the
second derivative of the non-edge ratio variation in Fig. 3(a) is
evaluated as shown in Fig. 3(b). The zero-crossing with a
threshold value of the curve in Fig. 3(b) is detected. After
removing the blurred images, the other frames can be used to
align the mosaic one.
(a) The amount of the non-edge pixels over frames;
(b) The second derivative of the curve in (a);
Fig. 3. The index for denoting the blurred images in an image sequence.
III. OPTIMAL SEAM PLANNING WITH A* ALGORITHM
During aligning the images, the dominant image frames
with plentiful image information are selected to be stitched first.
The images attempted to be stitched together should have the
similar overlap region on them. The SIFT features are
employed to estimate the overlap relation between image
frames and the homography matrix between them. The images
are then transferred to the coordinate of the mosaic image.
Directly cutting off the overlap region on the new image and
attaching it to the mosaic image will produce a rough edge in
the seam [6]. For smoothing the mosaic image and the new one
together, we have to utilize the overlap region between these
two images.
The simple method for smoothly aligning two images [6] is
to blend the color of these two images in the overlap region.
However, when there is motion parallax in the overlap region,
the blending the motion parallax commonly results in the
ghosting effect where an object has several observations in the
mosaic image. An efficient way of eliminating the ghosting
effect is to find a seam path [13] within the overlap region. The
aligned two images are cut off and blended along this seam.
Here, we propose an optimal seam planning algorithm by
utilizing the A* search algorithm to select the seam which
avoids the region of motion parallax, however, the image
observation around the region of motion parallax can still be
reserved at the same time. The optimal seam planning
algorithm is described in the following.
(a) The original image frames of the desired to be stitched ones.
(b) The left image has been transferred to the coordinate of the right one.
(c) The enhanced edge image.
(d) Remove the exclusive or of the two edge images in the overlap region.
Fig. 4. Aligning two images in the same coordinate.
Step. 0) After transferring the two images into the same
coordinate and aligning them together, the enhanced
edge image [12] is taken on each of them, such as Fig.
4(c).
Step. 1) Do the exclusive or operation in the overlap region of
the edge images as Fig. 4(d). The boundaries of
objects, which exactly overlays on those of the other,
would be neglected.
Step. 2) On the reserved edge images, evaluate the Hausdorff
216
Authorized licensed use limited to: UNIVERSITY OF SOUTHAMPTON. Downloaded on August 17,2020 at 19:53:20 UTC from IEEE Xplore. Restrictions apply.
distance from the edges of one image to the edge of
the other one. Remove the edges with longer distance
and obtain the remained edge image as Fig. 5(a).
Step. 3) Regard the remained edge image in the overlap region
as a 2D map with obstacles, like Fig. 5(a). The
selected optimal seam should avoid approaching these
edges which represents the region of motion parallax.
Dilate the regions of obstacles with a safe range
between the planned path to the obstacles.
Step. 4) Set the start point and several goal points initially
along the boundary of the overlap region, such as the
blue points in Fig. 5(b). If any of the start or goal
points locates inside the obstacle, it will be adjusted to
the place where is outside the obstacle and closest to
the original location. The decided start point and goal
points are labeled as the yellow ones in Fig. 5(b).
Step. 5) Utilize the A* search algorithm to find the shortest
path, which is the desired optimal seam, from the start
point and through the goal points sequentially.
The pixels in the overlap region, which are across the
selected seam, will be cut off. The overlap region between two
aligned images is then reduced to be a lane around the optimal
seam. Finally, the blending process will be performed along
this lane to smooth the color variations across these two aligned
images. Since the overlap regions that contain the same
contents on two images are decreased, the color blending
process does not have to fuse similar contents at different
locations. Hence, the ghosting effect can be greatly eliminated
by the proposed optimal seam selection.
(a) (b)
(c) Selected optimal seam (red path).
Fig. 5. Optimal seam path planning. The white contour indicates the boundary
of the overlap region between aligned images. (a) Remained edge image with
green color is regarded as the obstacle in the 2D map. (b) Start and goal points
assignment. (Initial: blue dots; Final: yellow dots)
IV. EXPERIMENTAL RESULTS
In the experiments, the mosaic images with blending the
whole overlap region and blending along the seam that is
selected by the A* search algorithm are compared here. The
two frames in Fig. 4(a) are stitched as one in Fig. 6. In Fig. 6(a),
we can see that the image blending without the seam selection
and the overlap reduction easily generates the ghosting effect at
several regions due to the motion parallax of the same object.
However, the image stitching with the proposed optimal seam
selection can eliminates these ghosting regions as shown in Fig.
6(b).
The frames with cluttered contents and complicated edges
are stitched as shown in Fig. 7. Since the camera are
continuously capturing during camera moving, the overlap
region between two consecutive frames is very wide. The
mosaic image with blending the whole overlap region
originates the ghosting effect throughout the edges in left
image of Fig. 7 (b). As shown the right image of Fig. 7(b), the
mosaic image with our proposed seam optimization algorithm
presents a stitched image with wide range and clear quality.
(a) (b)
Fig. 6. Comparisons of the mosaic images with two frames in Fig.4(a).
(a) The stitched image with blending the whole overlap region. The green
ellipses indicate the regions with ghosting effect.
(b) The stitched image with blending along the selected optimal seam.
(a) The original two image frames.
(b) Comparisons of the mosaic images. (Left/Right: without/with optimal
seam selecttion)
Fig. 7. Stitching the images with more cluttered contents.
V. CONCLUSION
The image stitching algorithm for eliminating the ghosting
effect is presented in this paper. The ghosting effect is
originated from the motion parallax that is generally happened
217
Authorized licensed use limited to: UNIVERSITY OF SOUTHAMPTON. Downloaded on August 17,2020 at 19:53:20 UTC from IEEE Xplore. Restrictions apply.
during camera moving. Eliminating the ghosting effect in
image stitching is actually to remove the blurred image region
and improve the image quality. Due to the vibration of camera
platform, the unclear ones are filtered out first by evaluating
the variation of histogram of edge between the consecutive
frames. The clear image frames are then transformed into the
same coordinate and aligned together. Here, the image
blending with the optimal seam selection is applied to reduce
the overlap region between images and eliminate the ghosting
effect. The inconsistent edges of two images in the overlap
region are considered as the obstacles, and the optimal seam
should avoid these inconsistent edges. The A* search algorithm
is employed to plan the seam path which can result in the
stitched image as a seamless mosaic one with clear image
resolution.
In the future, we will try to stitch the images from the front
and the bottom cameras of an unmanned air vehicle (UAV) at
the same time. The real-time image stitching algorithm will
also be developed and implemented on the UAV platform.
REFERENCES
[1] F. Caballero, L. Merinot, J. Ferruz, and A. Ollero, “Homography based
Kalman filter for mosaic building applications to UAV position
estimation,” in Proc. IEEE Conf. Robotics and Automation, pp. 10-14,
Apr. 2007
[2] M. Uyttendaele, A. Eden, and R. Szeliski. “Eliminating ghosting and
exposure artifacts in image mosaics,” in Proc. CVPR’01, vol. 2, pp.
509–516, 2001.
[3] H. Fang and J.C. Hart, “Textureshop: texture synthesis as a photograph
editing tool,” in Proc. ACM SIGGRAPH ’04, 2004.
[4] Q. Wu and Y. Yu, “Feature Matching and Deformation for
TextureSynthesis,” in Proc. ACM SIGGRAPH ’04, vol. 23, no. 3, pp.
362-365, 2004.
[5] D. Hasler and S. Susstrunk, “Colour handling in panoramic
photography,” in Proc. SPIE Electronic Imaging, vol. 4309, pp. 62-72,
Jan. 2001.
[6] A. Levin, A. Zomet, S. Peleg, and Y. Weiss, “Seamless image stitching
in the gradient domain,” in Proc. European Conf. Computer Vision, May
2004.
[7] J. Davis, “Mosaics of scenes with moving objects,” in CVPR ’98: in
Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 1998.
[8] A. Efros and W.T. Freeman, “Image quilting for texture synthesis and
transfer,” in Proc. ACM SIGGRAPH ’01, pp. 341-346, Aug. 2001.
[9] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick, “Graphcut
textures: image and video synthesis using graph cuts,” ACM Trans.
Graphics, in Proc. Int’l Conf. Computer Graphics and Interactive
Techniques (SIGGRAPH ’03), vol. 22, no. 3, pp. 277-286, July 2003.
[10] Yu Tang, Jungpil Shin, “De-ghosting for Image stitching with automatic
content-awareness,” in Proc. IEEE Conf. Pattern Recognition (ICPR),
pp.2210-2213, Aug. 2010
[11] T. Suzuki, Y. Amano, and T. Hashizume, “Vision based localization of a
small UAV for generating a large mosaic image,” in Proc. of SICE
Annual Conference, 2010.
[12] S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, P. Fua, and V.
Lepetit, “Gradient response maps for real-time detection of textureless
objects,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol.34,
no.5, pp.876-888, May 2012.
[13] J. Jia and C. K. Tang, “Image stitching using structure deformation,”
IEEE Trans. Pattern Analysis and Machine Intelligence, vol.30, no.4,
pp.617-631, 2008.