Improved Parallax Image Stitching Algorithm Based on Feature Block

Abstract

Image stitching aims at generating high-quality panoramas with the lowest computational cost. In this paper, we present an improved parallax image-stitching algorithm using feature blocks (PIFB), which achieves a more accurate alignment and faster calculation speed. First, each image is divided into feature blocks using an improved fuzzy C-Means (FCM) algorithm, and the characteristic descriptor of each feature block is extracted using scale invariant feature transform (SIFT). The feature matching block of the reference image and the target image are matched and then determined, and the image is pre-registered using the homography calculated by the feature points in the feature block. Finally, the overlapping area is optimized to avoid ghosting and shape distortion. The improved algorithm considering pre-blocking and block stitching effectively reduced the iterative process of feature point matching and homography calculation. More importantly, the problem that the calculated homography matrix was not global has been solved. Ghosting and shape warping are significantly eliminated by re-optimizing the overlap of the image. The performance of the proposed approach is demonstrated using several challenging cases.

1. Introduction

Stitching images is a broad intercrossing technology that is widely used in many disciplines, such as computer vision, image processing, and computer graphics, and it is widely used to obtain panorama images [1,2,3]. In recent years, it has become an active area of research in image processing technology and it plays an important role in many fields, such as minimally invasive surgical procedures, geological surveys, aerospace, and military reconnaissance [4]. At present, scale invariant feature transform (SIFT) [5] is the most popular algorithm in the field of image stitching. The features extracted by SIFT are invariant to image scale and rotation, and they are shown to provide robust matching across a substantial range of affine distortion, change in 3D viewpoint, addition of noise, and change in illumination. The stitching approaches based on this method usually estimate an optimal global homographic for each input image. A representative example is AutoStitch [6], as proposed by Brown et al., but global homographic methods work well for ideal cases when the camera translation is negligible or the scene is near planar [7,8].

Ghosting is generated when the input image has a large parallax. To achieve better stitching quality, Gao et al. assumed that the scene is composed of two predominating planes and proposed a dual homography warping (DHW) [9] model. It performs well for the specified scenes, but it cannot handle more complex conditions. Julio et al. proposed as-projective-as-possible warps (APAP) [10]. This method partitions the source image into many cell grids, and each cell grid is aligned through a homography warping that aims to be globally projective yet allows local non-projective deviations to account for violations in the assumed imaging conditions. The result is highly accurate image stitching, with significantly reduced ghosting effects. However, when dealing with two large parallax images, the homography matrix cannot handle the parallax and other techniques are needed to eliminate the ghost. Shape preserving half-projective warps (SPHP) [11] was proposed by Chang et al. This method proposed a novel parametric warp, the shape preserving half-projective warp, which is a spatial combination of a projective transformation and a similarity transformation. The projective transformation maintains good alignment in the overlapping region while the similarity transformation of the non-overlapping region keeps the original perspective of the image and reduces distortion. Fan Zhang presented a parallax-tolerant image stitching method (CPW) [12]. This method combines content-preserving and seam-driven warps, and position transform and shape distortion constraints are used to approximate the target image, which can minimize the registration error and preserve the stiffness of the scene. However, when there is a large translation and rotation between the original image and the target image, the stitching effect of the algorithm is greatly reduced owing to the limitation of the rigid constraint and blurring and ghosting occur [13,14,15].

In the existing algorithm, there are two disadvantages. First, the homography calculates feature points randomly selected from the set of feature matching points, and then it is judged whether most of the feature points satisfy the calculated homography transformation model [16,17,18]. If they are satisfied, the image is registered with this homography. If not, the matching feature points need to be re-selected to calculate the new homography. Iterations are performed through the above steps until most points satisfy the homography transformation model. A large amount of time is consumed by this method, and there is a case where the feature points are densely concentrated and the calculated homography is not global. Then, most existing methods use the SIFT method to calculate image feature points. In this algorithm, detecting and describing feature points in a full image and matching feature points are the two most time-consuming steps.

In order to solve the above problem, this study proposes an improved parallax image stitching algorithm based on feature blocks (PIFB). The picture is divided into several feature blocks through an improved fuzzy C-Means (FCM) algorithm, and the color information of the pixel and spatial distance of the pixels in the surrounding area are considered. In this method, the pixel points with common visual characteristics in the picture are effectively segmented into feature blocks. Then, feature points are detected by the SIFT algorithm based on feature blocks. The feature points in the matched feature blocks are used to calculate the global homography. The overlapping area is predetermined by feature segmentation of the image, avoiding feature point matching in non-overlapping areas. The reference image and the target image are pre-registered using a global homography, and then ghosting and shape distortion are eliminated by local optimization of the overlap region.

The remainder of this paper is organized as follows: Section 2 describes the image being segmented using an improved FCM algorithm to determine the overlap; Section 3 describes the global homography calculated using feature blocks. Section 4 describes the optimization algorithm for eliminating ghosts in overlapping parts; Section 5 demonstrates the results obtained using our approach, followed by a discussion and summary of our work in Section 6.

2. Feature Segmentation

2.1. Improved FCM Algorithm

The fuzzy C-Means (FCM) clustering algorithm was first introduced by Dunn [19] and later extended by Bezdek [20]. The algorithm is an iterative clustering method that produces an optimal partition by minimizing the weighted within group sum of the squared error objective function [21]. This method is widely used in the field of image feature segmentation in order to make better use of the position information of pixels in the image. The fuzzy local information C-Means (FLICM) [22] method is used in this article.

The objective function is defined as follows:

$$J_m={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{k=1}^c[u_{ki}^m{\Vert x_i-v_k\Vert }^2+G_{ki}]}}$$

(1)

where $x_i$ is the color value of the pixel, $v_k$ is the prototype of the center of cluster $k$, $u_{ki}$ is the degree of membership of $x_i$ in the $k\mathrm{th}$ cluster, $N$ is the number of pixels, $c$ is the number of clusters with $2\le c\le N$, and $m$ is the weighting exponent on each fuzzy membership.

It incorporates a fuzzy factor that considers both local space and gray information into its objective function. The fuzzy factor is defined in terms of

$$G_{ki}={\displaystyle \sum _{\begin{array}{l}j\in N_i\\ i\ne j\end{array}}\frac{1}{d_{ij}+1}{(1-u_{kj})}^m{\Vert x_j-v_k\Vert }^2}$$

(2)

where $v_k$ is the prototype of the center of the cluster, $d_{ij}$ is the spatial Euclidean distance between pixels $i$ and $j$, $u_{kj}$ is the degree of membership of the $j$th pixel in the $k$th cluster, and $m$ is the weighting exponent on each fuzzy membership. The $i$th pixel is the center of the local window, $k$ is the reference cluster, and the $j$th pixel belongs to the set of neighbors falling into a window around the $i$th pixel (N_i).

The two necessary conditions for $J_m$ to be at its local minimal extreme with respect to $u_{ki}$ and $v_k$ are obtained as follows:

$$u_{ki}=\frac{1}{{{\displaystyle \sum _{j=1}^c(\frac{{\Vert x_i-v_k\Vert }^2+G_{ki}}{{\Vert x_i-v_j\Vert }^2+G_{ji}})}}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$m-1$}\right.}}$$

(3)

$$v_k=\frac{{\displaystyle \sum _{i=1}^N{u}_{ki}^m{x}_i}}{{\displaystyle \sum _{i=1}^N{u}_{ki}^m}}$$

(4)

It is easy to see that the factor $G_{ki}$ combines in a fuzzy manner the spatial and color value information. The balance between the noise and image details is automatically achieved by the fuzzy local constraints. It enhances concurrently the clustering performance by the fact that almost all the other methods perform the clustering on a precomputed image while FLICM is applied on the original image, and thus, it can avoid loss of image details to a certain extent.

2.2. Determination of Clustering Parameters

The FLICM algorithm considers the local information of the pixel, but clustering results mainly depend on the initial conditions. If the number of clusters and the cluster center are not well selected, a bad segmentation result will occur. In order to improve the versatility and generalization ability of the algorithm, a method for automatically determining the number of clusters and the center of clustering is adopted in this study. The various steps of the method described in Figure 1 will be described in detail later.

First, the image to be split is converted from the RGB color space to the CIE Lab color space. As the three components of CIE Lab have different value ranges, they should be quantized, and the process of quantization is to specify the values of each dimension component. Then, the initial cluster number and cluster center are determined according to the following steps.

Step 1: The L, a, b components need to be quantized, and representative colors are quantized as color series to generate histograms with fewer histograms. Thus, in the process of finding histogram peaks, the number of algorithm cycles and execution time are reduced. Lab histogram is obtained.

Step 2: The initial cluster number of the FLICM and the initial cluster center are automatically determined by the hill-climbing method according to the Lab histogram of the color image to be segmented.

Step 3: Taking the non-zero histogram of the color histogram as a starting point.

1: Comparing the number of current rectangular prism pixels with the number of pixels of the adjacent rectangular column.

2: If the number of pixels of the adjacent histogram column and the current histogram column are different, move to the neighboring rectangular column with a larger number of pixels to climb the mountain.

3: If the adjacent histogram has the same number of pixels as the current histogram, the next adjacent histogram is searched. If the number of pixels is still the same, repeat the search for the adjacent rectangular column until the adjacent two square columns have different numbers of pixels, and the algorithm shifts to the neighboring rectangular column with the larger number of pixels to climb the mountain.

4: Steps 1–3 are repeated until a straight column that cannot climb the mountain is found, that is, when the number of pixels of the neighboring straight column is smaller than that of the current straight column, the current histogram is a peak (i.e., a local maximum).

Step 4: Take a non-zero histogram that has not been searched as a new starting point, go to Step 3 and get another peak until all the straight columns have been searched.

Step 5: These obtained peaks are the initial cluster centers of the input image, and the total number of these peaks is the number of clusters.

2.3. Feature Segmentation

On the basis that the number of clusters and the cluster center can be automatically determined, the parts with similar features in the image are divided. The detailed steps are as follows

Step 1: The number of clusters and the cluster center are obtained through the steps of Section 2.2.

Step 2: The algorithm end threshold is set to $\epsilon$, the number of iterations $b=1$, the weighted index is $m$, and the neighborhood window size of the pixel is w × w.

Step 3: The membership matrix $u_{ki}^b$ is obtained through Equation (3).

Step 4: According to Equation (4), the new cluster center $v_k^{(b+1)}$ is calculated using $u_{ki}^b$.

Step 5: If $\Vert V^{(b+1)}-V^{(b)}\Vert <\epsilon$, the algorithm ends, and the feature segmentation result is obtained; otherwise, $b=b+1$, and step 3 is performed.

Looking at image (a) of Figure 2, we can easily find several different regions of the image. If these regions with independent features can be divided before the image is stitched, the overlapping regions of the reference image and the target image can be easily determined. Image (b) shows the result of a picture being segmented. In the following sections we will describe how image stitching is done by feature segmentation.

3. Image Stitching Based on Feature Block

3.1. Feature Matching Block

By feature segmentation, the reference image and the target image are divided into different blocks, and each block is numbered as shown in Figure 3.

A new method for determining the overlapping regions of target images and reference images is used in this study. In the conventional method, feature points are detected in the full image, and in the process of matching the feature points, it is necessary to calculate the Euclidean distance of the feature points of the reference image and each feature point of the target image. However, image stitching is intended to stitch the overlapping portions of the reference image and the target image. Through the method of feature segmentation, the repeated calculation process of matching feature points is avoided, and the time complexity is reduced.

First, the feature blocks of the reference image and the target image are respectively obtained using the method outlined in the previous section and the feature points are detected by the SIFT method. Then, the number information of the feature block of the image is added to the feature points. Then, the feature matching block of each image block of the reference image is found. The detailed steps are as follows:

Step 1: The reference image $I_{nj}\left(n=1,2,\cdots ,n;j=1,2,\cdots ,N_n\right)$ and the target image $P_{mj}\left(n=1,2,\cdots ,m;j=1,2,\cdots ,N_n\right)$ are input, where $n$ and $m$ represent the number of feature blocks, and $nj$ represents $N_n$ feature points in the $n$-th feature block.

Step 2: All feature matching points of $I_{nj}$ are found from the target image through the SIFT algorithm, denoted as $I_{p^{\prime }j}^n$, where $p^{\prime }$ represents the feature matching block of the target image.

Step 3: The number of feature points in each $p^{\prime }$ is counted, and the image block matching the most feature points is selected. This is the feature matching block to be found, and the mismatch points in other image blocks are deleted. Taking Figure 3 as an example, it is assumed to be block 1’ in the target image. It is considered that the feature block 1 of the reference image matches this block of the target image, and the mismatch points with other blocks are deleted.

It is worth noting here that as the non-overlapping area of the reference image is still likely to have a very small number of matching feature points with the overlapping area of the target image, a minimum threshold $\phi$ is set to exclude the image block of the non-overlapping area matched with the image block of the overlapping area. Block 2 of the reference image is matched with the target image, except block 1’, and the matching block is determined. The remaining feature blocks in the reference image that match the target image are found by the above method.

The overlapping portion of the reference image and the target image is determined by the method of feature block matching. In the process of feature block matching, as the already matched feature blocks are excluded one by one, the time required for feature point matching is greatly reduced. Given that there is no matching feature point in the non-overlapping portion, there is no matching feature block, and thus, the overlapping area of the reference image and the target image is determined, and the feature blocks that are matched one by one are obtained.

3.2. Global Homography

In the traditional method, the feature matching points are randomly selected from the set of feature matching points to calculate the homography, and then it is judged whether most of the feature points satisfy the desired homography transformation model. If satisfied, the desired one is used and the homography registers the image. If it is not satisfied, the matching feature points need to be re-selected to calculate the homography transformation model, and the above steps are iteratively performed until most points satisfy the homography transformation model. However, with this algorithm for randomly selecting feature matching points, it will appear that the selected feature points are too concentrated. In this case, the calculated homography is not global, and only a few points satisfy the homography transformation model. Thus, the number of iterations of the algorithm is increased.

A new method for calculating the global optimal homography is proposed in this work. The homography is calculated by feature points that are selected from different feature matching blocks. The feature matching point set $P$ of the image is divided into $M$ different subsets according to the feature block, that is $P={\displaystyle \sum P_i\left(i=1,2,3\dots M\right)}$. In order to ensure that the feature matching point pairs used to calculate the homography matrix are global, $m$ feature point pairs are randomly selected from the $M$ subsets, respectively. We find that the algorithm has a greater calculation speed and registration accuracy when the total number of feature points selected each time does not exceed 10% of the total number of matching feature points. Global homography is expressed as

$$\left(\begin{array}{l}{x}_p\\ y_p\\ 1\end{array}\right)\sim H\left(\begin{array}{l}{x}_q\\ y_q\\ 1\end{array}\right)$$

(5)

where ${\left(x_p,y_p,1\right)}^T$ and ${\left(x_q,y_q,1\right)}^T$ are the homogeneous coordinates of a pair of feature matching points in the set $P$, $H$ is homography, and $H$ is expressed as

$$H=\left(\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right)$$

(6)

Direct linear transformation (DLT) [23] is a basic method to estimate $H$ from feature matching points. The calculation method is shown in Reference [23]. Randomly selecting feature matching points from different feature point matching subsets at random $k$ times, we can calculate $k$ global homography $H$. Through experiments, we find that when $k$ is between 10 and 15, the calculated homography has not only local but also global optimality.

Owing to misregistration, ghosting and other issues occur in overlapping areas. This study directly uses the overall registration of overlapping areas as a measure of splicing quality. We use the evaluation indicators proposed in the literature [24] to measure the quality of registration. The energy function is constructed by the normalized cross correlation (NCC) algorithm to represent the quality of registration. The smaller the energy value between image blocks, the higher the registration quality.

The overlapping between the reference image and the target image is determined according to the feature matching block in this algorithm, the non-overlapping region is deleted, and only the registration quality of the overlapping image block is calculated. Calculating the matching quality of the non-overlapping and overlapping regions of the image and the non-overlapping and non-overlapping regions is avoided here, which reduces the amount of calculation and error.

4. Local Adjustment Based on Grid Optimization

After pre-registering the reference image and the target image with the optimal global homography, it is necessary to optimize the alignment of the overlapping regions of the image, however, the unconstrained local alignment optimization will cause distortion and ghosting of the stitched image. Thus, based on the global pre-registration, the image is first meshed and then the local homography constraint, the local similarity constraint, and the global similarity constraint are introduced into the constructed energy function for mesh optimization.

Then, the total energy function is defined by

$$E\left(V\right)=E_{LH}\left(V\right)+{\lambda }_{LS}{E}_{LS}\left(V\right)+{\lambda }_{GS}{E}_{GS}\left(V\right)$$

(7)

where these three terms address the local homography, local similarity, and global similarity constraints. The set of vertices to be optimized is expressed as $V=\left\{{\widehat{v}}_j^{\left(i\right)}\right\}$, $\left(i=1,2,\dots ,n,j=1,2,\cdots m_i\right)$, where ${\widehat{v}}_j^{\left(i\right)}$ is a 2D coordinate point. It indicates the optimized position of the j-th mesh vertex of the i-th image on the canvas, and its position on the original image is represented as $v_j^{\left(i\right)}$ without a top mark. ${\lambda }_{LS}$ and ${\lambda }_{GS}$ are the adjustment weights of local and global similarities, respectively.

4.1. Local Homography Constraint

After the global optimal homography matrix transformation, the target image needs to be well aligned to the reference image. It is necessary to find a local optimal mesh transformation model that can move the feature points of the transformed target image to the feature point of the matched reference image. The local homography function is defined by

$$E_{LH}\left(V\right)={\displaystyle \sum _{\left(i,j\right)\in L}{\displaystyle \sum _{\left(p,q\right)\in M_{ij}}{\left|\widehat{p}-\widehat{q}\right|}_2^2}}$$

(8)

where $\widehat{p}$ and $\widehat{q}$ respectively represent the projection coordinates of the $p$ and $q$ feature point pairs on the canvas, and the projections of the matching feature points on the canvas are aligned by the optimization cost $\left|\widehat{p}-\widehat{q}\right|$. $\left(i,j\right)\in L$ means that $I_i$ and $I_j$ are a pair of registration pictures and $\left(p,q\right)\in M_{ij}$ represents that $p$ and $q$ are a pair of matching feature point pairs in the feature point pair set $M_{ij}$, obtained by performing feature matching from the picture $I_i$ to the picture $I_j$. Therefore, a local homography constraint optimization can also be called an alignment.

The target to be optimized comprises the mesh vertices, not the feature points. Thus, as shown in Figure 4, the feature points are expressed as a bilateral linear combination of the four vertices of their mesh: $p={\displaystyle \sum _{k=1}^4{\alpha }_k{v}_k}$, where ${\alpha }_k\left(k=1,2,3,4\right)$ represents the proportion of the area occupied by each one. For example, the closer $p$ is to $v_2$ and away from $v_3$, the larger ${\alpha }_2$ and the smaller ${\alpha }_3$. The transformed feature points are expressed as $\widehat{p}={\displaystyle \sum _{k=1}^4{\alpha }_k{\widehat{v}}_k}$, where the α_k (k = 1,2,3,4) transform remains unchanged.

4.2. Local Similarity Constraint

The role of the local similarity constraint is to ensure that each geometry transforms the geometric transformation of the overlapping region mesh over the entire mesh to ensure the continuity of the mesh transformation. The local similarity constraint function is defined by

$$E_{LS}\left(V\right)={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{\left(j,k\right)\in E_i}{\left|s_{jk}^{\left(i\right)}\left(v_k^{\left(i\right)}-v_j^{\left(i\right)}\right)-\left({\widehat{v}}_k^{\left(i\right)}-{\widehat{v}}_j^{\left(i\right)}\right)\right|}_2^2}}$$

(9)

where the optimization cost $\left|s_{jk}^{\left(i\right)}\left(v_k^{\left(i\right)}-v_j^{\left(i\right)}\right)-\left({\widehat{v}}_k^{\left(i\right)}-{\widehat{v}}_j^{\left(i\right)}\right)\right|$ indicates that the directed mesh edge vector $\left(v_k^{\left(i\right)}-v_j^{\left(i\right)}\right)$ should be aligned with the corresponding directed mesh edge vector $\left({\widehat{v}}_k^{\left(i\right)}-{\widehat{v}}_j^{\left(i\right)}\right)$ on the canvas after passing through the local similarity transformation $s_{jk}^{\left(i\right)}$, $s_{jk}^{\left(i\right)}$ represents the local similarity matrix, $E_i$ represents the set of mesh edges on image $I_i$, and $\left(j,k\right)\in E_i$ denotes that vertices $v_j^{\left(i\right)}$ and $v_k^{\left(i\right)}$ are the two vertices of the edge of the mesh.

According to the APAP method, the first column basis vector ${\left(c_{jk}^{\left(i\right)},s_{jk}^{\left(i\right)}\right)}^T$ of $s_{jk}^{\left(i\right)}$ can be expressed here as a combination of several vertices of the mesh edge $\left(j,k\right)$ field. The local similarity $s_{jk}^{\left(i\right)}$ is defined as the following formula:

$$s_{jk}^{\left(i\right)}=\arg \min {{\displaystyle \sum _{t=1}^6\left|s\left(v_{*t}^{\left(i\right)}-\frac{1}{2}\left(v_j^{\left(i\right)}+v_k^{\left(i\right)}\right)\right)-\left({\widehat{v}}_{*t}^{\left(i\right)}-\frac{1}{2}\left({\widehat{v}}_j^{\left(i\right)}+{\widehat{v}}_k^{\left(i\right)}\right)\right)\right|}}_2^2$$

(10)

where, $v_{*t}^{\left(i\right)}\left(t=1,2,\cdots 6\right)$ and ${\widehat{v}}_{*t}^{\left(i\right)}\left(t=1,2,\cdots 6\right)$ respectively represent the six vertices of the hexagon in which the mesh edge $\left(j,k\right)$ is located, as shown in Figure 5. The optimization cost means that the local similarity transformation with the center point of the mesh edge as the origin should maintain the shape structure of the local hexagon as much as possible.

Equation (10) can be solved by referring to the DLT method in the APAP algorithm.

4.3. Global Similarity Constraint

The role of the global similarity is that the mesh controlling the non-overlapping region is dominated by the global similarity transformation, while the mesh of the overlapping region is dominated by the alignment. This ensures that the overlapping regions can be fully aligned, and the non-overlapping region can try to ensure a similar transformation to reduce the projection distortion. The global similarity constraint function is defined by

$$E_{GS}={\displaystyle \sum _{i=1}^n{\displaystyle \sum _{\left(j,k\right)\in E_i}{\left({\omega }_{jk}^{\left(i\right)}\right)}^2{\left|\left({\widehat{v}}_k^{\left(i\right)}-{\widehat{v}}_j^{\left(i\right)}\right)-s_i\left(v_k^{\left(i\right)}-v_j^{\left(i\right)}\right)\right|}_2^2}}$$

(11)

where $s_i$ is obtained by the natural image stitching with the Global Similarity Prior (GSP) [25] method. The global similarity is controlled by ${\omega }_{jk}^{\left(i\right)}=d_{jk}^{\left(i\right)}+\beta$ through the distance $d_{jk}^{\left(i\right)}$ between the edge of the mesh and the overlapping region, and the adjustment amount $\beta$. As the edge $d_{jk}^{\left(i\right)}=0$ in the overlapping region, the $d_{jk}^{\left(i\right)}$ of the non-overlapping region becomes larger as it goes away from the overlapping region. Therefore, it can be realized that the alignment ability of local homography and local similarity is more important in the overlapping region, and the global homography is more important in the non-overlapping region.

4.4. The Energy Function Is Solved

The energy function is defined by the grid vertex set coordinates and each constraint function.

$$E\left(V\right)=E_{LH}\left(V\right)+{\lambda }_{LS}{E}_{LS}\left(V\right)+{\lambda }_{GS}{E}_{GS}\left(V\right)$$

(12)

Minimized results can be obtained by solving this sparse linear equation. ${\lambda }_{LS}$ and ${\lambda }_{GS}$ are two proportional parameters that are used to adjust the proportion of optimization, and here ${\lambda }_{LS}=0.5$ and ${\lambda }_{GS}=20$. All the items to be optimized are stored with a sparse matrix, the energy error is minimized by least squares, and the optimal vertex set $V$ is calculated. Finally, image mapping and pixel fusion are completed according to the obtained grid vertex coordinates.

Figure 6 shows the step-by-step process of the proposed algorithm.

5. Experiments

We compared our method with three other methods, AutoStitch, APAP, and CPW. The experiments were performed on a Windows 10 system with a 3.6 GHz CPU and 16 GB of RAM. The grid size was 40 × 40 for the mesh-based methods. The parameters involved in this study were as follows: the time window of FLICM was 3 × 3, the stacking threshold of FLICM was $\epsilon =0.001$, the weighting index of FLICM was $m=2$, and the number of global homography matrices was $k=15$. We tested the four methods on a large number of images. Four sets of images with increasing parallax were selected. There were lines and curves in these images, containing common scenes such as trees, buildings, cars, and squares.

The first experiment named School is shown in Figure 7. We first tested the performance of the algorithm using two images with a small parallax. In this figure, images (a) and (b) are the two images be stitched, (c) is the result of AutoStitch, (d) is the result of APAP, (e) is the result of CPW, and (f) is the result of PIFB. Observing the experimental results, we found that when the image disparity is relatively small, these algorithms can handle the image very well. Only a small part is not aligned, and the unaligned part is emphasized by the red wireframe and enlarged.

The second experiment named Temple is shown in Figure 8. The image parallax is increased. In this figure, images (a) and (b) are the images be stitched, (c) is the result of AutoStitch, (d) is the result of APAP, (e) is the result of CPW, and (f) is the result of PIFB. Observing the experimental results, through AutoStitch, although the images are well stitched together, we still find stitching errors in the detail texture, and the parts that are not aligned are marked by the red wireframe. The APAP algorithm makes the images well aligned by local alignment, but there are still misregistrations in some details. As the CPW algorithm only uses local alignment and similar transformation constraints, ghosting occurs in the stitching result because the constraints between the global grid blocks are not considered. However, with the algorithm (PIFB) proposed in this study, the images are pre-blocked and then aligned, and global similarities and local similarities are considered, hence, it can better eliminate ghosting and shape distortion.

The third experiment named Building is shown in Figure 9. The parallax of the picture is further increased. In this figure, images (a) and (b) are the two pictures to be stitched, (c) is the result of AutoStitch, (d) is the result of APAP, (e) is the result of CPW, and (f) is the result of PIFB. Observing the experimental results, through AutoStitch, although the images are well stitched together, we still find stitching errors in the detail texture, and the parts that are not aligned are marked by the red wireframe. Although the AutoStitch algorithm has performed very well, there are still parts that are not aligned, such as the chair highlighted by the red wireframe. The APAP algorithm makes aligns the images well by local alignment, but there are still misregistrations in some details. The CPW algorithm uses local and similar alignment, but unaligned railings still appear in the image. However, with the algorithm (PIFB) proposed in this study, the images are pre-blocked and then aligned, and global similarities and local similarities are considered so it can better eliminate ghosting and shape distortion.

The last experiment, named Square, is shown in Figure 10. Very large parallax images are used to test the stability of the algorithm. In this figure, images (a) and (b) are the two images to be stitched, (c) is the result of AutoStitch, (d) is the result of APAP, (e) is the result of CPW, and (f) is the result of PIFB. Observing the experimental results, AutoStitch does not have the ability to stitch effectively, and the parts that are not aligned are marked by the red wireframe. The APAP algorithm is still not satisfactory in the processing of details. The CPW algorithm deals with large parallax pictures and due to the global similarity constraints, large-scale ghosts appear in the registration results. However, with the algorithm (PIFB) proposed in this study, the images are pre-blocked and then aligned, and global and local similarities are considered, and therefore, some stability is still obtained with the challenge of a large parallax.

In the visual representation of the picture stitching results, the algorithm proposed in this work has good stability and the ability to stitch large parallax pictures. In terms of the objective evaluation, the root mean square error used by the APAP algorithm is used as the registration criterion to measure the objective index of each algorithm. The calculation formula is

$$RMSE\left(f\right)=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N{\Vert f\left(x_i\right)-x_i^{\prime }\Vert }^2}}$$

(13)

where $f$ is the transformation relationship between the two images and ${\left\{x_i,x_i^{\prime }\right\}}_{i=1}^N$ is the matched feature point pair. The RMSE represents the difference between the two, and the smaller the RMSE value is, the better the result of image stitching. As the commercial software AutoStitch cannot calculate its RMSE, this study only takes into account the RMSE values of several commonly used algorithms. The RMSE of the different images is the average of ten calculations, and the details are presented in

Table 1. RMSE and time-consuming comparison of different algorithms.

Algorithms	School		Temple		Building		Square
	RMSE	Time/s	RMSE	Time/s	RMSE	Time/s	RMSE	Time/s
AutoStitch		5.122		3.86		9.996		10.32
APAP	15.221	8.521	11.125	8.066	15.965	16.470	12.102	19.012
CPW	16.621	7.351	15.801	6.125	16.021	15.332	15.916	17.256
PIFB	15.356	8.423	8.895	8.036	12.227	18.021	9.653	18.995

.

Table 1 shows that the CPW algorithm has a poor stitching effect; although it takes the shortest time, its RMSE is the largest. Compared with the CPW algorithm, the APAP algorithm has a significantly reduced RMSE, but the time consumption is significantly greater. Compared with the CPW algorithm, the proposed algorithm has an increase in time consumption because the feature segmentation performed before stitching consumes a certain amount of time, but the RMSE value decreases significantly and the stitching effect is greatly enhanced. Compared with the APAP algorithm, the RMSE value and time consumption are substantially reduced. In summary, the proposed algorithm dominates the stitching effect, and even though the stitching effect is greatly improved, the computing time it needs is less than that of APAP and only a little higher than that of CPW. Although the RMSE value of the AutoStitch algorithm cannot be compared in Table 1, we can see from Figure 10 that as the parallax increases, the stitching performance of the AutoStitch algorithm decreases significantly. Thus, the running time does not represent the advantage of the AutoStitch algorithm. Therefore, through a comprehensive comparison, the proposed algorithm is advantageous.

6. Conclusions

An effective and efficient image stitching method based on feature blocks is proposed in this study. First, each image will be divided into feature blocks using an improved FCM algorithm, and the feature block information is added to the SIFT descriptor. Image stitching is a process of splicing a group of pictures with overlapping parts into a panoramic picture, and the splicing of overlapping parts involves merging areas with the same features in this part. Therefore, the overlapping area of the picture can be quickly determined by dividing the picture into different feature blocks with independent attributes. Second, the homography is globalized by the feature block to calculate the homography between the reference image and the target image, and the image is pre-registered through this homography. Finally, the local homography constraint, the local similarity constraint, and the global similarity constraint are introduced into the constructed energy function for mesh optimization, and the ghost and distortion of the overlap are eliminated. The stitching quality, flexibility, and computational efficiency of our method are carefully evaluated by several comparative experiments on a series of challenging cases. Our future work will concentrate on more robust occlusion handling and improved stitching speed.