A Novel and Effective Cooperative RANSAC Image Matching Method Using Geometry Histogram-Based Constructed Reduced Correspondence Set

Abstract

The success of many computer vision and pattern recognition applications depends on matching local features on two or more images. Because the initial correspondence set—i.e., the set of the initial feature pairs—is often contaminated by mismatches, removing mismatches is a necessary task prior to image matching. In this paper, we first propose a fast geometry histogram-based (GH-based) mismatch removal strategy to construct a reduced correspondence set $C^{reduced,GH}$ from the initial correspondence set $C^{ini}$. Next, we propose an effective cooperative random sample consensus (COOSAC) method for remote sensing image matching. COOSAC consists of a RANSAC, called $RANSA{C}^{ini}$ working on $C^{ini}$, and a tiny RANSAC, called $RANSA{C}^{tiny,GH}$ working on a randomly selected subset of $C^{reduced,GH}$. In $RANSA{C}^{tiny,GH}$, an iterative area constraint-based sampling strategy is proposed to estimate the model solution of $C^{tiny,GH}$ until the specified confidence level is reached, and then $RANSA{C}^{ini}$ utilizes the estimated model solution of $C^{tiny,GH}$ to calculate the inlier rate of $C^{ini}$. COOSAC repeats the above cooperation between $RANSA{C}^{tiny,GH}$ and $RANSA{C}^{ini}$ until the specified confidence level is reached, reporting the resultant model solution of $C^{ini}$. For convenience, our image matching method is called the GH-COOSAC method. Based on several testing datasets, thorough experimental results demonstrate that the proposed GH-COOSAC method achieves lower computational cost and higher matching accuracy benefits when compared with the state-of-the-art image matching methods.

1. Introduction

Remote sensing images have been applied in many applications, such as disaster assessment, construction site monitoring, building change detection, military applications, and so on. To extend the areas that remote sensing images can cover, image matching has received extensive attention. In the image matching field, establishing reliable feature correspondences for two or more images is a challenging issue, and it has many applications in stitching, registration, retrieval, object recognition, and 3D reconstruction. Prior to image matching, using the difference-of-Gaussian (DOG) technique, contrast checking strategy, and Hessian matrix-based testing rule, Lowe’s scale-invariant feature transform (SIFT) [1] is used to extract the feature point sets of the source image $I_s$ and that of the target image $I_t$, respectively. Furthermore, the initial correspondence set $C^{ini}$ between the two feature point sets, i.e., the set of the initial feature pairs, is constructed such that for each correspondence $c_i$ in $C^{ini}$, 1<= i <= $|C^{ini}|$, the 128-dimensional feature vector of the left feature point of $c_i$ is more similar to that of the right feature point of $c_i$ than the other right feature points in $C^{ini}$. The initial correspondence set $C^{ini}$ is usually contaminated by mismatches, i.e., outliers. In what follows, given an initial correspondence set $C^{ini}$, we introduce the related works for removing mismatches and image matching.

1.1. Related Works

Given $C^{ini}$, Fischer and Bolles [2] proposed a pioneer image matching work, called the random sample consensus (RANSAC) method, to determine the homography transformation between two feature point sets in $C^{ini}$. Usually, the determined homography transformation is also called the model solution of $C^{ini}$. The detailed definition of the model solution of $C^{ini}$ will be given in Section 3.1. The RANSAC method first randomly selects four correspondences, i.e., four feature-pairs, from $C^{ini}$ to estimate the model solution as a hypothesis, and then using the estimated model solution, it counts the number of fitting correspondences and verifies the inlier rate of $C^{ini}$, denoted by $I{R}^{ini,current}$, where the inlier rate of $C^{ini}$ is defined to be the number of fitting correspondences over the size of $C^{ini}$. Next, RANSAC repeats the above hypothesize-and-verify (HAV) process, and then it uses the newly estimated model solution to calculate the new inlier rate of $C^{ini}$, denoted by $I{R}^{ini,new}$. If $I{R}^{ini,current}<I{R}^{ini,new}$, the previous inlier rate $I{R}^{ini,current}$ is replaced by the new inlier rate $I{R}^{ini,new}$; otherwise, we repeat the above HAV process until the specified confidence level is reached. Here, the confidence level of RANSAC is defined to be a probability value $p=1-{(1-{\left(I{R}^{ini,current}\right)}^4)}^k$, where k denotes the number of iterative HAV processes required in RANSAC. The resultant model solution of $C^{ini}$ thus serves as the established correspondence between two feature point sets in $C^{ini}$. Although RANSAC is robust, it usually requires huge computational cost under low inlier rate cases.

Capel [3] proposed a bail-out test approach to accelerate RANSAC. In his approach, first, a correspondence subset $C^{sub}$ is constructed by randomly selecting a subset of $C^{ini}$. Using one model solution of $C^{sub}$, if the inlier rate of $C^{sub}$ is significantly less than the current inlier rate of $C^{ini}$, the subset $C^{sub}$ and the estimated model solution are discarded; otherwise, the inlier rate of $C^{sub}$ is larger than the current inlier rate of $C^{ini}$ and the estimated model solution of $C^{sub}$ is used to calculate the inlier rate of $C^{ini}$. As a result, the bail-out test approach could reduce the average computational cost involved in the verification of each hypothesis, improving the computational cost performance of RANSAC. However, this approach might incorrectly reject good hypotheses; on the other hand, it might suffer from the accuracy degradation.

Instead of treating all correspondences equally in RANSAC, Chum and Matas [4] proposed a progressive sample consensus method to assign different sampling priorities of all correspondences in $C^{ini}$ by using the ratio test strategy. Hence, their method is robust against a large number of correspondences with low inlier rate. The lower the correspondence’s priority is, the later the correspondence participates the HAV process. For example, at the beginning of their method, the four correspondences with top four priorities are first selected as the hypothesis to calculate the inlier rate of $C^{ini}$. In the same argument, the four correspondences with slightly lower priorities are selected as the hypothesis in the next iteration. Under low inlier rate cases, their method achieved good robustness. However, due to the priority-based selection rule, the four correspondences with low priorities are often discarded after reaching the termination condition, leading to the matching accuracy degradation.

Matas and Chum [5] proposed a randomized RANSAC method to accelerate RANSAC. In their method, a small subset $C^{small}$ is first constructed from $C^{ini}$. Once all correspondences in $C^{small}$ are all inliers, the estimated model solution of $C^{small}$ is used to calculate the inlier rate of the remaining correspondences in $C^{ini}-C^{small}$ where the operator “-” denotes the set difference operator. Their method achieved better computation performance relative to RANSAC and the bail-out test approach [3]. However, there is room for improvement on the matching accuracy performance.

Due to the simplicity and robustness of RANSAC, the above-mentioned related works [3,4,5] are inspired by RANSAC. In [6], Raguram et al. provided a comprehensive overview of the RANSAC-like methods and introduced a universal RANSAC framework to extend the HAV process of RANSAC by considering more practical and computational issues, such as prefiltering unreliable correspondences, performing sampling check, etc. Consequently, it brings out more aspects in the image matching issue.

Ma et al. [7] proposed an effective vector field consensus (VFC) method to establish a reduced correspondence set $C^{reduced}$ from $C^{ini}$. In VFC, the reduced correspondence set is determined based on the coherence of the underlying motion fields rather than the geometric constraints. The inlier rate of $C^{ini}$ is expressed as the rate of the size of $C^{reduced}$, denoted by $|C^{reduced}|$, over $|C^{ini}|$, accelerating the matching performance and preserving the accuracy performance. Using the classical density-based spatial clustering method of applications with noise [8], Jiang et al. proposed a robust feature matching (RFM) method [9] to detect groups of matches with similar motion. In RFM, the correspondence subset with inconsistent motion features is discarded. Setting the inlier rate of $C^{ini}$ to the rate of $|C^{reduced}|$ over $|C^{ini}|$, RFM can also accelerate the matching performance and preserve the accuracy performance.

Ma et al. [10] proposed a locality preserving (LP) matching method to prune correspondences in $C^{ini}$ if they violate the spatial neighborhood relationship. Using Ma et al.’s method [10] as a preprocessing step to create a reduced correspondence set $C^{reduced}$, Wang et al. [11] randomly sampled four correspondences from $C^{reduced}$, and then calculated the estimated model solution of $C^{reduced}$. Further, using the estimated model solution, the HAV process is performed on $C^{reduced}$ to obtain the inlier rate of $C^{reduced}$. They repeated the above model solution estimation process for $C^{reduced}$ until the specified confidence level is reached. Finally, the resultant model solution of $C^{reduced}$ is used to calculate the inlier rate of $C^{ini}$, achieving execution time and accuracy improvements. For convenience, Wang et al.’s method is called the LP-RANSAC method.

Inspired by the image segmentation and registration methods in [12,13,14], Liu et al. [15] proposed a motion consistency- and correspondence growing-based method to reduce the correspondence set by using a coarse-to-fine strategy. Wang and Chen [16] proposed a guided local outlier factor (GLOF) method to filter out abnormal correspondences. In GLOF, a tentative correspondence set is first constructed by matching multi-features. Next, mismatches are identified and removed by using the anomaly detection technique and a user-defined threshold, creating a robust reduced correspondence set $C^{reduced}$. Expressing the inlier rate of $C^{ini}$ as the rate of $|C^{reduced}|$ over $|C^{ini}|$, GLOF can enhance the matching accuracy performance.

1.2. Contributions

In this paper, for remote sensing matching, we propose a new cooperative RANSAC (COOSAC) method using a geometry histogram-based (GH-based) constructed reduced correspondence set. For convenience, the proposed method is called the GH-COOSAC method, as depicted in Figure 1, and its contributions are clarified below.

• Differing from the previous correspondence set reduction methods, we propose a fast GH-based mismatch removal strategy to construct a reduced correspondence set $C^{reduced,GH}$ from $C^{ini}$ such that for all correspondences in $C^{reduced,GH}$, the orientation- and length-consistency can be preserved.
• Differing from the related image matching works introduced in Section 1.1, we propose a GH-COOSAC method which consists of a RANSAC, called $RANSA{C}^{ini}$ working on $C^{ini}$, and a tiny RANSAC, called $RANSA{C}^{tiny,GH}$ working on a tiny correspondence set $C^{tiny,GH}$ that is randomly constructed from a small subset of $C^{reduced,GH}$. In addition, an iterative area constraint-based sampling strategy is deployed into $RANSA{C}^{tiny}$ to estimate the model solution of $C^{tiny,GH}$ until the specified confidence level is reached. Then, $RANSA{C}^{ini}$ utilizes the estimated model solution H to calculate the inlier rate of $C^{ini}$. We repeat the above cooperative HAV processes between $RANSA{C}^{tiny,GH}$ and $RANSA{C}^{ini}$ until the specified confidence level is reached, reporting the best model solution of $C^{ini}$. The advantages of the proposed method are low computational cost, in particular, under the inlier rate interval [0.5, 0.9], and high matching accuracy benefits when compared with the other methods [2,7,9,11,16].
• Based on several testing datasets with different inlier rates, thorough experimental results justify that the proposed GH-COOSAC method achieves low computational cost and high matching accuracy benefits relative to RANSAC [2] and the four state-of-the-art methods, namely VFC [7], RFM [9], LP-RANSAC [11], and GLOF [16].

The remainder of this paper is organized as follows. In Section 2, the proposed GH-based mismatch removal strategy is presented to construct a reduced correspondence set $C^{reduced,GH}$. In Section 3, the proposed GH-COOSAC method is presented. In Section 4, thorough experiments are carried out to demonstrate low computational cost and high matching accuracy benefits of the proposed method. In Section 5, some concluding remarks are addressed.

2. The Proposed Geometry Histogram-Based (GH-Based) Mismatch Removal Strategy to Construct a Reduced Correspondence Set

Based on the initial correspondence set $C^{ini}$, this section first presents the proposed fast GH-based mismatch removal strategy to construct a reduced correspondence set $C^{reduced,GH}$. Next, the computational complexity of the proposed GH-based strategy is analyzed. Here, each image-pair is assumed to have mainly positional transformation with negligible rotational differences.

2.1. Reconstructing the Reduced Correspondence Set $C^{reduced,GH}$

Considering every correspondence $c_i(=[(x_i,y_i),(x_i^{\prime },y_i^{\prime })])$ in $C^{ini}$, $1\le i\le |C^{ini}|$, the angle of $c_i$ is expressed as

$$\begin{array}{cc}\hfill Angle\left(c_i\right)& =ta{n}^{-1}\frac{y_i^{\prime }-y_i}{x_i^{\prime }-x_i}\hfill \end{array}$$

(1)

respectively, where $0\le Angle\left(c_i\right)\le {180}^{\circ }$. Here, the term “angle” and the term “orientation” denote the same thing. In addition, the length of $c_i$ is expressed as

$$\begin{array}{c}\hfill Length\left(c_i\right)=\sqrt{{(x_i^{\prime }-x_i)}^2+{(y_i^{\prime }-y_i)}^2}\end{array}$$

(2)

Based on the orientations of all correspondences in $C^{ini}$ which are obtained from one image-pair in the dataset “Airport” (https://www.sensefly.com/education/datasets/?dataset=1419, accessed on 6 April 2022), Figure 2a illustrates the quantized orientation histogram of $C^{ini}$, namely $H\left(\theta \right)$, where the x-axis denotes the quantized orientation $\theta$, where the size of one bin occupies five degrees, and the y-axis denotes the frequency. In $H\left(\theta \right)$, we select the correspondence subset in the highest peak and its left and right $k_1$ neighboring peaks as a tentative reduced correspondence set $C^{\theta ,k_1}(\subset C^{ini})$. Figure 2b demonstrates the reduced quantized orientation histogram $H(\theta ,k_1)$ for $k_1=1$. Consequently, we have filtered out the orientation-inconsistency correspondences in $C^{ini}$, but retain the relatively orientation-consistent correspondences in $C^{ini}$.

Based on the lengths of all correspondences in Figure 2b, Figure 2c illustrates the corresponding quantized length histogram of Figure 2b, where the length of one bin in the histogram occupies 20 pixels. Further, we select the correspondence subset in the highest peak and its left and right neighboring peaks of Figure 2c, namely $C^{\theta ,k_1;L,k_2}$ where “L” denotes the quantized length, as the reduced correspondence set $C^{reduced,GH}$. Figure 2d demonstrates the histogram of the reduced correspondence set $C^{reduced,GH}$ for $k_2=1$. As a result, we collect all correspondences from Figure 2d as the reduced correspondence set $C^{reduced,GH}$ in which all correspondences are relatively orientation- and length-consistent.

Figure 3a depicts the testing image-pair used in Figure 2. In Figure 3b, the initial correspondence set $C^{ini}$ of the testing image-pair is shown by those 2643 lines. Based on the quantized orientation histogram in Figure 2b, Figure 3c illustrates the 1390 lines of the reduced correspondence set $C^{\theta ,k_1}$ for $k_1$ =1. Figure 3d illustrates the reduced correspondence set $C^{reduced,GH}$ with $|C^{reduced,GH}|=392$ by using the proposed GH-based mismatch removal strategy.

2.2. Computational Complexity Analysis

From the description of the proposed GH-based mismatch removal strategy to construct the reduced correspondence set $C^{reduced,GH}$, by Equation (1), it takes $O\left(1\right)$ time to calculate the orientation of every $c_i\in C^{ini}$. Therefore, it takes $O\left(\right|C^{ini}\left|\right)$ time to construct the orientation histogram of the initial correspondence set $C^{ini}$.

By Equation (2), the length of every $c_i\in C^{\theta ,1}$ can be calculated in $O\left(1\right)$ time. Therefore, for all correspondences in $C^{\theta ,1}$, their lengths can be calculated in $O\left(\right|C^{\theta ,1}\left|\right)$ time. Furthermore, it takes $O\left(\right|C^{\theta ,1;length,1}\left|\right)$ time to construct the reduced correspondence set $C^{reduced,GH}$. Because $|C^{\theta ,1;length,1}|<<|C^{\theta ,1}|<<|C^{ini}|$, totally, it takes $O\left(\right|C^{ini}\left|\right)$ time to construct the reduced correspondence set $C^{reduced,GH}$ from $C^{ini}$.

Proposition 1.

Given an initial correspondence set $C^{ini}$, the reduced correspondence set $C^{reduced,GH}$ can be constructed in $O\left(\right|C^{ini}\left|\right)$ time, i.e., linear-time, where $|C^{reduced,GH}|<<|C^{ini}|$, by using the proposed GH-based Mismatch Removal Strategy.

3. The Proposed GH-COOSAC Method for Image Matching

Given an initial correspondence set $C^{ini}$, using the GH-based mismatch removal Strategy mentioned in Section 2, suppose the reduced correspondence set $C^{reduced,GH}$ has been constructed. In this section, the proposed GH-COOSAC method for image matching is presented. Initially, a tiny correspondence set $C^{tiny,GH}$ is constructed by randomly selecting a subset of the reduced correspondence set $C^{reduced,GH}$, where $|C^{tiny,GH}|<<|C^{reduced,GH}|$ and in our experiment, we set $|C^{tiny,GH}|=0.2\ast |C^{reduced,GH}|$.

To provide a more readable introduction, we sketch the logic behind the main steps of the proposed GH-COOSAC method. In the GH-COOSAC method, COOSAC consists of a RANSAC, called $RANSA{C}^{ini}$ which works on $C^{ini}$, and a tiny RANSAC, called $RANSA{C}^{tiny,GH}$ which works on $C^{tiny,GH}$. In $RANSA{C}^{tiny,GH}$, an iterative area constraint-based sampling strategy is proposed to estimate the model solution of $C^{tiny,GH}$ until the specified confidence level, say 99.5%, is reached. Let the estimated model solution of $C^{tiny,GH}$ be denoted by H. Then, $RANSA{C}^{ini}$ utilizes the estimated model solution H to calculate the inlier rate of $C^{ini}$. COOSAC repeats the above cooperation between $RANSA{C}^{tiny,GH}$ and $RANSA{C}^{ini}$ until the specified confidence level of $RANSA{C}^{ini}$, say 99.5%, is reached. Finally, the resultant model solution of $C^{ini}$ is reported.

3.1. The HAV Process in $RANSA{C}^{tiny,GH}$

As depicted in the hypothesis stage of $RANSA{C}^{tiny,GH}$ in Figure 4, the proposed area constraint-based sampling strategy first randomly selects four correspondences from $C^{tiny,GH}$, and then it examines whether the area of every two selected correspondences is larger than the specified threshold $T_{area}$. Empirically, $T_{area}$ is set to 1000 pixels. In Figure 5, the area surrounded by two selected correspondences is depicted. If the calculated area of every two selected correspondences is larger than $T_{area}$, $RANSA{C}^{tiny,GH}$ proceeds to solve the model solution of $C^{tiny,GH}$ by using the four selected correspondences; otherwise, the selected four correspondences are dropped, and we select another four correspondences from $C^{tiny,GH}$ again.

Suppose the four selected correspondences are denoted by $c_1$=$((x_1,y_1),(x_1^{\prime },y_1^{\prime }))$, $c_2$ = $((x_2,y_2),(x_2^{\prime },y_2^{\prime }))$, $c_3$ = $((x_3,y_3),(x_3^{\prime },y_3^{\prime }))$, and $c_4$ = $((x_4,y_4),(x_4^{\prime },y_4^{\prime }))$. Among the four selected correspondences, selecting any two correspondences as one combination, there are six combinations in total. For each combination, denoted by $(c_i,c_j)$ for $i\ne j$, the area of the quadrilateral surrounded by $c_i$ and $c_j$ can be calculated by

$$\begin{array}{c}\hfill Area(c_i,c_j)=\left|\frac{x_1\ast y_2+x_2\ast y_2^{\prime }+x_2^{\prime }\ast y_1^{\prime }+x_1^{\prime }\ast y_1-y_1\ast x_2-y_2\ast x_2^{\prime }-y_2^{\prime }\ast x_1^{\prime }-y_1^{\prime }\ast x_1}{2}\right|\end{array}$$

(3)

In the six combinations, if there exists one combination $(c_i,c_j)$ whose area is less than the specified threshold $T_{area}$, it indicates that the four selected correspondences, $c_1$, $c_2$, $c_3$, and $c_4$, are invalid and should be discarded. Figure 5 illustrates one invalid case because the area surrounded by $c_2$ and $c_3$ is less than the threshold, so the four selected correspondences are discarded. Otherwise, if the area of every combination is larger than or equal to $T_{area}$, the four selected correspondences are valid and can be used to estimate the model solution of $C^{tiny,GH}$.

Then, we put the coordinates of the four valid correspondences, $c_1$, $c_2$, $c_3$, and $c_4$, into the following homography equation:

$$\begin{array}{c}\hfill \left[\begin{array}{c}{x}_i^{\prime }\\ y_i^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{21}& 1\end{array}\right]\left[\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right]\end{array}$$

(4)

where $((x_i,y_i),(x_i^{\prime },y_i^{\prime }))=c_i$ for $1<=i<=4$. By using the SVD (singular value decomposition) method [17] to solve Equation (4), the eight homography parameters, $h_{00}$, $h_{01}$, $h_{02}$, $h_{10}$, $h_{11}$, $h_{12}$, $h_{20}$, and $h_{21}$, can be solved, and the solution is used as the estimated model solution of $C^{tiny,GH}$.

As depicted in the verification stage of Figure 4, we utilize the estimated model solution of $C^{tiny,GH}$ to calculate the inlier rate of $C^{tiny,GH}$, denoted by $I{R}^{tiny,GH,new}$. Let $I{R}^{tiny,GH,current}$ denote the highest inlier rate of $C^{tiny,GH}$ so far. If $I{R}^{tiny,GH,current}<I{R}^{tiny,GH,new}$, $I{R}^{tiny,GH,current}$ is replaced by $I{R}^{tiny,GH,new}$; otherwise, the value of $I{R}^{tiny,GH,current}$ is retained.

The above HAV process of $RANSA{C}^{tiny,GH}$ is repeated until the confidence level of $RANSA{C}^{tiny,GH}$, denoted by $Pr{o}^{tiny,GH}$, is larger than the specified confidence level. Here, $Pr{o}^{tiny,GH}$ is defined by

$$\begin{array}{c}\hfill Pr{o}^{tiny,GH}=1-{(1-{\left(I{R}^{tiny,GH,current}\right)}^4)}^k\end{array}$$

(5)

where the parameter k denotes the number of iterations required by $RANSA{C}^{tiny,GH}$. For example, suppose $I{R}^{tiny,GH,current}=$ 0.6 and the iteration number spent is $k=32$, the value of $Pr{o}^{tiny,GH}$ can achieve at least 99% $(=1-{(1-{\left(0.6\right)}^4)}^{32})$. In our implementation, as demonstrated in Section 4, the specified confidence level $T_{Pr{o}^{tiny,GH}}$ is set to 99.5%.

3.2. The Verification Process in $RANSA{C}^{ini}$

Suppose in the first round, the number of the HAV iterations used in $RANSA{C}^{tiny,GH}$ is $k_1$ and the confidence level, $Pr{o}^{tiny,GH}$, reaches the specified threshold $T_{Pr{o}^{tiny,GH}}$. In the proposed COOSAC mechanism, once $RANSA{C}^{ini}$ receives the estimated model solution of $RANSA{C}^{tiny,GH}$, namely H, it views H as the hypothesis of $C^{ini}$ directly. Then, in the verification stage of $RANSA{C}^{ini}$, $RANSA{C}^{ini}$ utilizes the hypothesis H to calculate the first inlier rate of the initial correspondence set $C^{ini}$, denoted by $I{R}^{ini,current}$. The confidence level of $RANSA{C}^{ini}$ can be expressed as

$$\begin{array}{c}\hfill Pr{o}^{ini}=1-{(1-{\left(I{R}^{ini,current}\right)}^4)}^{k_1}\end{array}$$

(6)

However, after performing $k_1$ HAV iterations on $C^{tiny,GH}$, the resultant model solution of $RANSA{C}^{tiny,GH}$, might be trapped into a local optimization. In order to jump out of the trap, in the second round, we create a new tiny correspondence set from the reduced correspondence set $C^{reduced,GH}$ again. By the same argument, the HAV process of $RANSA{C}^{tiny,GH}$ is repeated $k_2$ times until its confidence level reaches the specified confidence level. Then, the newly resultant model solution of $RANSA{C}^{tiny,GH}$ is utilized by $RANSA{C}^{ini}$ to calculate the new inlier rate of $C^{ini}$, denoted by $I{R}^{ini,new}$. If $I{R}^{ini,current}<I{R}^{ini,new}$, the value of $I{R}^{ini,current}$ is replaced by that of $I{R}^{ini,new}$; otherwise, the value of $I{R}^{ini,current}$ is retained. After completing the second round, the confidence level of $RANSA{C}^{ini}$ can be expressed as

$$\begin{array}{c}\hfill Pr{o}^{ini}=1-{(1-{\left(I{R}^{ini,current}\right)}^4)}^{k_1+k_2}\end{array}$$

(7)

Following the same argument, suppose the cooperative HAV process between $RANSA{C}^{tiny,GH}$ and $RANSA{C}^{ini}$ is repeated m rounds. That is, $RANSA{C}^{tiny,GH}$ is constructed m times and totally, $k_1$, $k_2$, $\dots$, and $k_m$ iterations are required to obtain the m estimated model solutions of the randomly constructed m tiny correspondence sets. $RANSA{C}^{ini}$ utilizes the received m estimated model solutions to obtain m inlier rates of $C^{ini}$, and finally, it selects the model solution with the highest inlier rate, namely $I{R}^{ini,current}$, as the resultant model solution. Consequently, the final confidence level of $RANSA{C}^{ini}$ can be expressed as

$$\begin{array}{c}\hfill Pr{o}^{ini}=1-{(1-{\left(I{R}^{ini,current}\right)}^4)}^{k_1+k_2+\dots +k_m}\end{array}$$

(8)

At this time, the value of $Pr{o}^{ini}$ in Equation (8) achieves the specified confidence level, e.g., 99.5%.

3.3. Computational Complexity Analysis and Benefit

From the above discussion, in the proposed GH-COOSAC method, we know that overall, $RANSA{C}^{ini}$ receives m different model solutions from $RANSA{C}^{tiny,GH}$, and $RANSA{C}^{ini}$ takes $O(m\ast |C^{ini}\left|\right)$ time to complete all verification processes on $C^{ini}$. In addition, $RANSA{C}^{tiny,GH}$ takes $O(m\ast |C^{tiny,GH}\left|\right)$ time to construct m tiny correspondence sets from the reduced correspondence set $C^{reduced,GH}$, and it takes $O((k_1+k_2+\dots +k_m)\ast |C^{tiny,GH}\left|\right)$ time to obtain these m model solutions to be used by $RANSA{C}^{ini}$. Totally, the proposed GH-COOSAC method takes $O(m\ast |C^{ini}\left|\right)+O(m\ast |C^{tiny,GH}\left|\right)+O((k_1+k_2+\dots +k_m)\ast |C^{tiny,GH}\left|\right)$ time to complete the whole image matching task. Because $|C^{tiny,GH}|<<|C^{ini}|$, $O(m\ast |C^{ini}\left|\right)+O(m\ast |C^{tiny,GH}\left|\right)$ can be simplified to $O(m\ast |C^{ini}|$). Consequently, we have the following result.

Proposition 2.

The proposed GH-COOSAC method takes $O(m\ast |C^{ini}\left|\right)+O((k_1+k_2+\dots +k_m)\ast |C^{tiny,GH}\left|\right)$ time to complete the image matching task.

Under the confidence level, i.e., 99.5%, Proposition 2 indicates that the proposed GH-COOSAC method takes $O(m\ast |C^{ini}\left|\right)+O((k_1+k_2+\dots +k_m)\ast |C^{tiny,GH}\left|\right)$ time for the image matching task. Because the inlier rate of $C^{ini}$ is less than the inlier rate of of $C^{tiny,GH}$, the traditional RANSAC method $RANSA{C}^{ini}$ need take at least $O((k_1+k_2+\dots +k_m)\ast |C^{ini}\left|\right)$ time to reach the same confidence level, 99.5%, for the image matching task. Because practically, the two conditions, namely $(m<<k_1+k_2+\dots +k_m)$ and $\left(\right|C^{tiny,GH}|<<|C^{ini}\left|\right)$, hold, it yields

$$\begin{array}{c}\hfill O(m\ast |C^{ini}\left|\right)+O((k_1+k_2+\dots +k_m)\ast |C^{tiny,GH}\left|\right)<O((k_1+k_2+\dots +k_m)\ast |C^{ini}\left|\right)\end{array}$$

(9)

Equation (9) implies the following result.

Proposition 3.

Under the same confidence level, the computational cost of the proposed GH-COOSAC method is lower than the RANSAC method.

In the next section, thorough experimental data will be used to justify the low computational cost and high matching accuracy benefits of the proposed GH-COOSAC method relative to RANSAC [2]. In Section 4.3, a discussion will be provided to explain why under the inlier rate interval [0.5, 0.9], the computational cost of the proposed method is lower than the state-of-the-art methods in [7,9,11,16].

4. Experimental Results

Three public remote sensing correspondence datasets, namely “Airport”, “Small Village”, and “University Campus” which are downloaded from the website: https://www.sensefly.com/education/datasets/ (accessed on 6 April 2022), used to evaluate the execution time and accuracy performance of the proposed GH-COOSAC method, RANSAC, and the four state-of-the-art methods, namely VFC [7], RFM [9], LP-RANSAC [11], and GLOF [16]. To measure the average accuracy performance of each considered image matching method, three accuracy metrics, namely precision, recall, and F1-score, are used.

For fairness, all comparative methods and the proposed method have been implemented on a computer with an Intel Core i7-4790 CPU 3.6 GHz and 24 GB RAM. The operating system is the Microsoft Windows 10 64-bit operating system. The program development environment is Visual Studio 2019. The execution time (in seconds) required in each considered method is measured by running the execution code of that method. The C++ source code of RANSAC [2] can be accessed from OpenCV; the C++ source code of VFC [7] can be accessed from the website https://github.com/jiayi-ma/VFC (accessed on 6 April 2022). The Matlab source codes of the RFM method [9] (see website https://github.com/StaRainJ/RFM-SCAN) (accessed on 6 April 2022) and the GLOF method [16] (see website https://github.com/gwang-cv/GLOF (accessed on 6 April 2022))are transformed to the C++ source codes. In the LP-RANSAC method [11], the Python source code of the LP method [10] accessed from the website https://github.com/jiayi-ma/LPM_Python (accessed on 6 April 2022) has been transformed to the C++ source code. The C++ source code of our GH-COOSAC method can be accessed from the website https://github.com/YCccTseng/GH-COOSAC (accessed on 6 April 2022). In the next subsection, the configuration of the proposed method is described.

4.1. The Parameter Setting of the Proposed GH-COOSAC Method

For each dataset, five testing image-pairs are selected to compare the performance for the considered methods. Overall, fifteen testing image-pairs as well as their initial correspondence sets are used and they can be accessed from the website: https://github.com/YCccTseng/GH-COOSAC (accessed on 6 April 2022). Each initial correspondence set $C^{ini}$, in the next paragraph, we explain how to create the nine different correspondence sets, each set containing a ground truth inlier sets, for the nine inlier rates, 0.1, 0.2, 0.3, …, and 0.9, respectively. For example, for the inlier rate 0.1, the rate of the created ground truth inlier set over the created correspondence set equals 0.1.

In one initial correspondence set $C^{ini}$, we first randomly sample four correspondences ${10}^8$ times, and then ${10}^8$ inlier sets are produced by using the traditional RANSAC method. In the ${10}^8$ inlier sets, we select the inlier set with the maximal inlier rate, say $R_{inlier}$, as the initial ground truth inlier set. Based on the specified inlier rate $r_{inlier}$$\in \{0.1,0.2,\dots ,0.9\}$, if $r_{inlier}$ > $R_{inlier}$, we discard the outliers from the initial ground truth inlier set gradually until the specified inlier rate is reached; otherwise, if $r_{inlier}$ < $R_{inlier}$, we discard the inliers from the initial ground truth inlier set gradually until the specified inlier rate is reached, producing the ground truth inlier set with the specified inlier rate.

In our implementation, the size of the tiny correspondence set $C^{tiny,GH}$ follows the constraint: $|C^{tiny,GH}|=0.2\ast |C^{reduced,GH}|$. The specified confidence level is set to $99.5\%$ to both $RANSA{C}^{tiny,GH}$ and $RANSA{C}^{ini}$. As described in Section 3.2, according to the specified confidence level, the value of m and the values of $k_1$, $k_2$, …, and $k_m$ are automatically determined by the proposed GH-COOSAC method. For simplicity, the proposed GH-COOSAC method is also called “Ours”.

4.2. Statistical Accuracy Comparison

To compare the accuracy performance among the considered methods, the above-mentioned three metrics are used. Let true positive (TP), true negative (TN), false positive (FP), and false negative (FN) be defined by the number of correspondences that are correctly verified as inliers, that are correctly verified as outliers, that are erroneously verified as inliers, and that are erroneously verified as outliers, respectively. The three accuracy metrics, namely precision, recall, and F1-score are defined by

$$\begin{array}{c}\hfill precision=\frac{TP}{TP+FP}\\ \hfill recall=\frac{TP}{TP+FN}\\ \hfill F1-score=\frac{2}{\frac{1}{\frac{1}{precision}+\frac{1}{recall}}}\end{array}$$

(10)

In Equation (10), the value of “precision” indicates the rate of the number of correspondences that are correctly verified as inliers over the sum of the number of correspondences that are correctly verified as inliers and the number of correspondences that are erroneously verified as inliers. The value of “recall” indicates the rate of the number of correspondences that are correctly verified as inliers over the sum of the number of correspondences that are correctly verified as inliers and the number of correspondences that are erroneously verified as outliers. F1-score indicates the precision-recall tradeoff, and the value of “F1-score” can be viewed as an overall matching accuracy performance.

Based on the “Airport” dataset, Figure 6a–c illustrate the precision, recall, and F1-score performance comparison of the six considered methods, respectively. From Figure 6a, we observe that the proposed GH-COOSAC method, RANSAC, and LP-RANSAC have clearly better precision performance relative to RFM, VFC, and GLOF. From Figure 6b,c, under low inlier rate cases, the proposed method has good recall and F1-score performance, but under middle inlier cases, the proposed method just achieves fair recall and F1-score performance. Figure 6b,c indicate that under high inlier cases, VFC, RFM, and GLOF have clearly better recall and F1-score performance.

As mentioned in Equation (10), the metric “F1-score” is used to measure the precision-recall tradeoff of each considered image matching method. To provide the average quantitative F1-score performance of each method, three inlier rate intervals, namely [0.1, 0.5], [0.5, 0.9], and [0.1, 0.9], are considered. Based on the “Airport” dataset,

Table 1. Average F1-score comparison for the “Airport” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.939	0.871	0.888	0.918	0.840	0.964
[0.5, 0.9]	0.919	0.987	0.979	0.917	0.975	0.936
[0.1, 0.9]	0.930	0.925	0.931	0.917	0.903	0.949

illustrates the average F1-score performance comparison for the three inlier rate intervals. From Table 1, we observe that in the interval [0.1, 0.5], the proposed method has the best F1-score performance, namely $0.964$ in boldface. In the interval [0.5, 0.9], VFC has the best F1-score performance, namely $0.987$ in boldface, but RFM and GLOF are quite competitive with VFC. In the complete interval [0.1, 0.9], the proposed method has the best F1-score performance, namely $0.949$ in boldface.

Based on the “Small Village” and “University Campus” datasets, Figure 7a–c and Figure 8a–c demonstrate the precision, recall, and F1-score performance comparison of the six considered methods, respectively.

Table 2. Average F1-score comparison for the “Small Village” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.940	0.885	0.898	0.920	0.769	0.962
[0.5, 0.9]	0.943	0.987	0.983	0.913	0.977	0.933
[0.1, 0.9]	0.940	0.932	0.939	0.916	0.865	0.947

and

Table 3. Average F1-score comparison for the “University Campus” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.966	0.906	0.879	0.954	0.798	0.973
[0.5, 0.9]	0.958	0.986	0.980	0.951	0.981	0.958
[0.1, 0.9]	0.961	0.943	0.926	0.952	0.882	0.965

tabulate the average quantitative F1-score performance of each method for the three inlier rate intervals, namely [0.1, 0.5], [0.5, 0.9], and [0.1, 0.9] for the “Small Village” dataset and the “University Campus” dataset, respectively. It comes to the same conclusion that in the interval [0.1, 0.5], the proposed method has the best F1-score performance in boldface. In the interval [0.5, 0.9], VFC has the best F1-score performance in boldface, but RFM and GLOF are quite competitive with VFC. In the complete interval [0.1, 0.9], the proposed method has the best F1-score performance in boldface.

4.3. Execution Time Comparison Furthermore, Discussion

Based on the three testing datasets, the average execution time (in seconds) required in each considered method is illustrated in Figure 9, where the x-axis denotes the ground truth inlier rate and the y-axis denotes the required average execution time.

4.3.1. Execution Time Comparison

Table 4. Average execution time (in seconds) comparison for the “Airport” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.515	0.027	2.320	0.028	0.294	0.232
[0.5, 0.9]	0.004	0.010	0.303	0.009	0.108	0.001
[0.1, 0.9]	0.287	0.019	1.398	0.019	0.207	0.129

,

Table 5. Average execution time (in seconds) comparison for the “Small Village” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.484	0.056	10.689	0.056	0.636	0.220
[0.5, 0.9]	0.003	0.022	1.681	0.021	0.248	0.002
[0.1, 0.9]	0.270	0.040	6.554	0.040	0.454	0.123

and

Table 6. Average execution time (in seconds) comparison for the “University Campus” dataset.

Inlier Rate Interval	RANSAC [2]	VFC [7]	RFM [9]	LP-RANSAC [11]	GLOF [16]	Ours
[0.1, 0.5]	0.438	0.061	12.339	0.061	0.708	0.222
[0.5, 0.9]	0.003	0.024	1.902	0.023	0.278	0.001
[0.1, 0.9]	0.244	0.044	7.548	0.043	0.505	0.124

tabulate the average execution time performance for the “Airport” dataset, the “Small Village” dataset, and the “University Campus” dataset, respectively. From Table 4, Table 5 and Table 6, we observe that under the two inlier rate intervals, namely [0.1, 0.5] and [0.1, 0.9], VFC and LP-RANSAC require the least execution time, and the proposed GH-COOSAC method is ranked third. Under the inlier rate interval [0.5, 0.9], the proposed method requires the least execution time and is ranked first.

4.3.2. Discussion

Proposition 3 indicates that the proposed GH-COOSAC takes less computational cost relative to RANSAC [2]. No matter what the inlier rate interval is, Table 4, Table 5 and Table 6 always justified the low computational cost merit of the proposed GH-COOSAC method relative to RANSAC.

Equation (5) implies that to reach the specified confidence level, 99.5 %, for high inlier rate cases, the probability of the event, namely $I{R}^{tiny,GH,current}<I{R}^{tiny,GH,new}$, is often higher than that for low inlier rate cases in each round. On the other hand, under higher inlier rate cases, the proposed GH-COOSAC method takes less iterations in each round than that under low inlier rate cases. This can explain why in Table 4, Table 5 and Table 6, under the inlier rate interval [0.5, 0.9], the proposed method achieves a clearly low computational cost benefit.

In summary, Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6 indicate that the proposed GH-COOSAC method achieves low computational cost and high matching accuracy benefits when compared with RANSAC [2] and the four state-of-the-art methods [7,9,11,16].

5. Conclusions

The proposed GH-COOSAC method for remote sensing image matching has been presented. In GH-COOSAC, the computational complexity analysis of the GH-based mismatch removal strategy has been provided to show the low computational cost benefit of the proposed strategy. In addition, a detailed computational complexity comparison between the proposed GH-COOSAC method and the traditional RANSAC method is given to show the computational cost superiority of the proposed method. No matter what the inlier rate interval is, experimental results always justify the low computational cost merit of the proposed method relative to RANSAC. In Section 4.3.2, the detailed discussion is provided to explain why under the inlier rate interval [0.5, 0.9], the proposed method achieves a clearly low computational cost benefit relative to all comparative methods. Generally, thorough experimental results demonstrate that the proposed GH-COOSAC method achieves low computational cost and high matching accuracy benefits relative to RANSAC [2] and the four state-of-the-art image matching methods [7,9,11,16].