Feature Point Tracking Method for Visual SLAM Based on Multi-Condition Constraints in Light Changing Environment

Abstract

In scenes where there are lighting changes, localization may fail for visual SLAM due to feature point tracking failure. Thus, a feature point tracking method based on multi-condition constraints is proposed for visual SLAM. The proposed method tracks the feature points of optical flow from aspects such as the overall motion position of feature points, descriptor grayscale information, and spatial geometric constraints. First, to solve the problem of feature point mismatch in complex environments, we propose a feature point mismatch removal method that combines optical flow, descriptor, and RANSAC. We eliminate incorrect feature point matches layer by layer through these constraints. The uniformity of feature point distribution in the image can then affect the accuracy of camera pose estimation, and different scenes can also affect the difficulty of feature point extraction. In order to balance the quality and uniformity of the extracted feature points, we propose an adaptive mask homogenization method that adaptively adjusts the mask radius according to the quality of feature points. Experiments conducted on the EuRoC dataset show that the proposed method which integrates the improved feature point mismatch removal method and mask homogenization method into feature point tracking, exhibits robustness and accuracy under various interferences such as lighting changes, image blurring, and unclear textures. Compared to the RANSAC method, we reduce the location error by about 85% using the EuRoC dataset.

1. Introduction

Simultaneous Localization and Mapping (SLAM) is an essential task in robotics and computer vision that aims to simultaneously estimate a robot’s pose and construct a map of the environment [1]. As images and videos can provide detailed information about the environment, a significant portion of SLAM research focuses on visual SLAM (VSLAM) [2]. Feature point tracking is a critical element in VSLAM since it has a significant impact on the accuracy of localization and mapping in later stages. The classic algorithms for feature point tracking include the Lucas–Kanade algorithm [3,4], BRIEF algorithm [5], SIFT algorithm [6], SURF algorithm [7] and ORB algorithm [8,9]. The classic algorithms for feature point tracking can be divided into the feature point descriptor method [10,11,12,13] and optical flow method [14,15,16,17]. The feature point descriptor method involves extracting feature points from two image frames, computing their descriptors, and matching the feature points based on the similarity of their descriptors to achieve feature point matching, such as SIFT [6], SURF [7], ORB [8], and BRIEF [5]. Furthermore, the optical flow method determines the direction and speed of each pixel’s movement on the image plane by calculating the change in pixel gray value between adjacent frames, such as dense optical flow [18,19,20] and sparse optical flow [21,22,23].

In the process of feature point tracking, to improve visual SLAM positioning accuracy, it is necessary to eliminate mismatched point pairs caused by environmental light changes, motion blur, noise, and other factors. For the feature point descriptor, brute force matching is a common method used [24]. In this method, each feature point in one image is compared with every feature point in the other image to find the best match. This is achieved by calculating a distance measure between the feature descriptors. The feature point with the smallest distance is considered to be the best match. Although brute force matching is very simple, the initial brute force matching contains many false matching results. This is because brute force matching only considers that the two descriptors are the most similar to other descriptors, and does not consider that the feature points in image A are not the same point in image B. Thus, brute force matching will match each feature point in image A with a feature point in image B, regardless of whether the matching point exists. Therefore, some researchers use the global distance threshold to reduce the result of mismatching. Only when the distance between two feature point descriptors is less than the set threshold are the two feature points considered matching point pairs. Nonetheless, there are notable variations in the distinguishability of feature point descriptors across different scenarios, necessitating ongoing debugging and threshold determination. Furthermore, dealing with feature points that have inconsistent distinguishability is also a challenging task [25]. Cross-validation is a common method in brute force matching [26]. For the feature point $p_i$ in image A, the most similar feature point in image B is $q_i$, and for $q_i$, the most similar feature point in image A is also $p_i$, that is, the two feature points are the most similar to each other, so it is considered that $(p_i,q_i)$ is a pair of matching points. Although cross-validation can reduce the mismatch of brute force matching to a certain extent, the effect is limited. RANSAC [27] can effectively remove mismatching points; however, RANSAC also removes some correct matching points, reducing the number of matching point pairs [28], which has a negative impact on the subsequent position and attitude estimation due to these erroneous judgements.

For the optical flow method, reverse optical flow is a common method [29,30]. The idea of reverse optical flow is very simple. Similar to cross-validation, the error of matching point bidirectional optical flow is used to remove the mismatching point pair. Reverse optical flow can effectively reduce the inaccurate tracking of optical flow tracking and obtain correct tracking matching results. However, its shortcomings are also evident. First, the optical flow method is only suitable for slow-motion scenes. Secondly, the reverse optical flow is used to eliminate mismatches, which is equivalent to performing optical flow twice in the entire tracking and matching process, resulting in increased time consumption. RANSAC is another common method. However, some correct matching points will also be removed with RANSAC, which is unacceptable when the image quality is not high and the number of feature points tracked is small. Moreover, in similar areas where the image is blurred, RANSAC will fit some mismatched points.

Overall, the descriptor can describe the feature points well, and can effectively eliminate the mismatched points in the case of blurred images, but it cannot eliminate the mismatched points with high regional similarity. The optical flow method reduces the computational complexity required for feature point matching and improves feature point tracking accuracy. However, the optical flow method requires that the camera move slowly and is susceptible to changes in light. In an environment with changing lighting conditions, where the image quality is drastically reduced, a single mismatch point pair elimination method is not effective in eliminating erroneous matches. Therefore, we propose a multi-constraint feature point mismatch removal method. By utilizing the complementary features among various constraints, we can more stably eliminate incorrect matching pairs. Firstly, feature point pairs are extracted using the optical flow method. Then, the BRIEF descriptor of the optical flow point pairs is calculated, and erroneous matches are eliminated through a global threshold method. Subsequently, RANSAC is used to further remove mismatched pairs. Furthermore, we use optical flow errors and feature descriptor errors to avoid correct matching point pairs eliminated by RANSAC. Finally, to retain high-quality matching points, we propose an adaptive image mask homogenization method that adaptively selects and retains the feature points extracted by optical flow tracking. The main contributions of this paper are summarized as follows:

• In order to solve the problem of feature point mismatch, this paper proposes a feature point mismatch removal method that combines optical flow, descriptor, and RANSAC, suitable for complex environments.
• We propose an improved mask homogenization method to optimize the quality of feature points and improve their distribution in the image.
• We integrate the proposed feature point mismatch removal method and improve mask homogenization method in feature tracking. Experimental results on EuRoC [31] dataset demonstrate the robustness and accuracy of our method in various complex environments.

2. Materials and Methods

2.1. Overall Flow

The flow of the proposed multi-constraint feature point tracking is shown in Figure 1. Firstly, we use the optical flow method to match feature point pairs. Then, the BRIEF descriptors of the match feature point pairs are calculated, and a global threshold is used to eliminate matching point pairs with low similarity. RANSAC is then used to eliminate the mismatches feature point pairs that cannot be eliminated by BRIEF descriptor matching. To address potential misjudgments, we evaluate the quality of matching points using optical flow error and descriptor matching error. Matching points of high quality will not be eliminated. Finally, an adaptive mask homogenization method is used to homogenize the feature points, and again extract a sufficient number of feature points.

2.2. Optical Flow Tracking

Optical flow tracking is the first step in the multi-constraint feature point tracking. We use Lucas–Kanade optical flow method for the base of feature point tracking. For a pixel located at $(x,y)$ at time t, it is moved to $(x+dx,y+dy)$ at time $t+dt$. If the brightness of the pixels in the image is the same, the intensity I of the pixel located at $(x,y)$ is preserved as follows:

$$I(x+dx,y+dy,t+dt)=I(x,y,t)$$

(1)

If the time difference between adjacent image frames is short, Taylor series expansion to the left of Equation (2) can be used to obtain:

$$I(x+dx,y+dy,t+dt)\approx I(x,y,t)+\frac{\partial I}{\partial x}dx+\frac{\partial I}{\partial y}dy+\frac{\partial I}{\partial t}dt$$

(2)

By combining Equations (1) and (2) and divided by $dt$, we obtain the following equations:

$$\frac{\partial I}{\partial x}\frac{dx}{dt}+\frac{\partial I}{\partial y}\frac{dy}{dt}=-\frac{\partial I}{\partial t}dt$$

(3)

where $\frac{dx}{dt}$ and $\frac{dy}{dt}$ represent the velocity on the $x,y$ axis and can be denoted as u and v. By denoting $\frac{\partial I}{\partial x}$ to $I_x$, $\frac{\partial I}{\partial y}$ to $I_y$ and the change of grayscale value with time as $I_t$, Equation (3) can be rewritten more compactly in a matrix form as:

$$\left[\begin{array}{cc}{I}_x& I_y\end{array}\right]\left[\begin{array}{c}u\\ v\end{array}\right]=-I_t$$

(4)

Assuming that feature points within a certain size have the same motion, we can select several feature points to form linear equations. Then, we can calculate the u and v of the feature points in that region, and obtain the positions of the feature points in the adjacent frames.

2.3. BRIEF Descriptor

When the image quality is not high in a light-changing environment, some mismatch point pairs remain after optical flow. To eliminate the mismatch point pairs, the BRIEF descriptors of the remaining feature points are calculated, and the matching point pairs with low similarity are eliminated using global threshold. BRIEF descriptor is a binary local feature descriptor used for image matching and target recognition. The BRIEF descriptor operates by randomly sampling local regions of an image and comparing the grayscale values of the sampled points. Based on the comparison results, a binary string is generated to serve as a descriptor for the region. The descriptor is centered around a feature point and encompasses a neighborhood window of size $S\times S$. Within this window, N pairs of random points are selected (usually $N=256$), and the grayscale values of each pair are compared. The resulting comparison values are converted into binary codes, which are concatenated to form the BRIEF descriptor for the feature point.

2.4. RANSAC

For some mismatching point pairs that the BRIEF descriptor cannot exclude, we use the RANSAC method to exclude. Step 1: Randomly select four groups of points from matching points, calculate the parameter of the homography matrix H as Formula (5) shown:

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=H\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(5)

where $(x,y)$ is the feature point position of the previous frame, $(x^{\prime },y^{\prime })$ is the feature point position of the next frame, H is the homography matrix. Step 2: Judge whether the other point pairs belonging to the homography matrix H (the error is less than a given threshold value), record the number of data belonging to the homography matrix H; Step 3: Assess the number of points in the second part, if it is the biggest, then retain the homography matrix H, if not, abandon the homography matrix H; Step 4: Repeat step 1, 2, and 3 until the error probability P is smaller than 0.01, then the homography matrix H is found; Step 5: After model found, estimate the homography matrix H again.

2.5. Preserve Feature Points

For the problem that RANSAC will eliminate some correct matching points, we use optical flow error and descriptor matching error to evaluate the quality of these matching points:

$$\frac{e_{flow}}{N_{flow}}+\frac{e_{brief}}{N_{brief}}<\tau$$

(6)

where, $e_{flow}$ is the optical flow tracking error, $e_{brief}$ is the matching error of BRIEF descriptor, $N_{flow}$ and $N_{brief}$ are the corresponding normalization factor, $\tau$ is the quality threshold, default set to 1. When the matching point pair deleted by RANSAC method satisfies Equation (6), the matching point pair is retained. The entire processing process removes mismatches from the optical flow results in terms of motion position, grayscale gradient (descriptor), and geometric constraints (RANSAC), and preserves as many correct matching points as possible.

2.6. Feature Point Homogenization

After deleting the mismatched point pairs, the number of feature points tracked by the optical flow has decreased. To achieve accurate and stable feature point tracking, it is necessary to supplement the reduced number of feature points. To avoid excessive concentration or sparsity of the supplemented feature points, we use image masking to uniformly distribute the feature points. The initially extracted feature points are shown in Figure 2a. First, the feature points tracked by optical flow in previous frame are sorted in descending order of tracking frequency, and an image mask with the same size as the image is created, which each pixel set to 255. The value of the mask position corresponding to the feature point is then traversed. If the value is 255, this feature point is retained. Furthermore, the area of a circle with the feature point position as its center and r as its radius is set to 0 in image mask. If the value of the image mask position corresponding to each feature point is 0, the feature points are deleted. After all the feature points have been traversed, the result of preserving the uniformly distributed feature points is shown in Figure 2b, and the final image mask is shown in Figure 2c. Finally, the Shi-Tomasi feature points are extracted in the image corresponding to the non-zero region of the image mask, as shown by the blue points in Figure 2d.

The distance parameter r of the feature point is a crucial factor in achieving image mask homogenization. If r is set too small, the uniformity of the image will be poor, leading to the extraction of too many feature points. Conversely, if r is set too large, the quality of feature points extracted in low-texture environments may be compromised, although it can make the distribution of feature points more uniform. In order to automatically adapt to changes in the environment, we dynamically adjust the feature point distance r based on the quality of the optical flow feature points. We classify the tracked feature points into high-quality feature points and low-quality feature points, where the quality of the feature point is evaluated based on the variance $\sigma$ of N pixels surrounding the feature point, as shown in following:

$${\sigma }^2=\frac{1}{N}\sum _{k=0}^{N-1}{(I_k-I_{mean})}^2$$

(7)

where $I_{mean}$ is the grayscale average of N pixels, $I_k$ is the pixel value of the k pixel point. When $\sigma$ exceeds the threshold value, the feature point is considered a high-quality feature point, otherwise it is a low-quality feature point. Then, we dynamically adjust r by counting the proportion of low quality feature points $\alpha$:

$$r=\left\{\begin{array}{cc}max(r\times k,\frac{R}{2}),\hfill & \alpha >{\tau }_1\hfill \\ min(\frac{r}{k},2R),\hfill & \alpha <{\tau }_2\hfill \\ r,\hfill & otherwise\hfill \end{array}\right.$$

(8)

where k is the adjustment coefficient, which defaults to 0.8; R is the original radius, which defaults to 20; ${\tau }_1$ is the upper limit threshold of low quality feature points, and the default value is 0.4, ${\tau }_2$ is the lower threshold value of low quality feature points, and the default value is 0.03. When the proportion of low quality feature points is high, reduce the feature point distance r to obtain more close high-quality feature points; When the proportion of high-quality feature points is very high, increase the feature point distance r to make the distribution of feature points more uniform. When the scale is appropriate, the current feature point distance r is maintained unchanged.

3. Results

This section presents a series of comparison tests conducted on the current popular visual-inertial datasets collected on-board a micro aerial vehicle (EuRoC) to demonstrate the superiority and effectiveness of the proposed method. The datasets contain synchronized stereo images, IMU measurements and accurate ground truth and is divided into three scenarios: Machine Hall, Vicon Room 1, and Vicon Room 2. Each scenario is further divided into three levels: simple, medium, and difficult. Machine Hall is a factory workshop with five sequence ($MH01-MH05$). There is a dark environment in the difficulty level $MH04$ and $MH05$ images. There are three sequence ($V101-V103$) in the Vicon Room 1. There are image shading changes in images sequence of the medium level $V102$ and the difficulty level $V103$. The Vicon Room 2 has three sequences, $V201$, $V202$, and $V203$. Among these, the medium level $V202$ and the difficult level $V203$ sequences have issues with image blurring caused by rapid dithering and slight dark environments. The experimental platform consists of (1) AMD Ryzen 5 4600H CPU with a frequency of 3.0 G/Hz, and 4 GB memory; (2) G++ compiler as the software tool under Ubuntu 18.04. We tested and validated the proposed algorithm based on the open-source VINS-Mono framework. We first perform an ablation study to show the effect of each module of our proposed method. Then we conducted comparative experiments with other methods. In Formula (6): $N_{flow}=30$, $N_{brief}=60$ are used for the experiments. In Formula (7): $\sigma =20$ are used for the experiments.

3.1. Ablations Study

We conducted ablation studies to demonstrate the effectiveness of each module in our proposed method. We evaluate the performance of optical flow tracking, BRIEF descriptor, RANSAC, preserve feature points and feature point homogenization. Because the algorithm is based on sparse optical flow, detecting and tracking too many feature points will consume more time. So the number of tracked feature points is limited to 260. In this experiment, the root mean square error (RMSE) of the absolute trajectory error (APE) of positioning is used to evaluate the effect of mismatch removal. The EVO evaluation tool is used to evaluate the positioning results. The quantitative comparison results are shown in

Table 1. Location error (RMSE/m) of each step on EuRoC dataset, ave. and std. are the average and the standard deviation of location error. a is only the module of optical flow tracking; b is the module of optical flow tracking and BRIEF descriptor; c is the module of optical flow tracking, BRIEF descriptor and RANSAC; d is the module of optical flow tracking, BRIEF descriptor, RANSAC and preserve feature points; e is the module of optical flow tracking, BRIEF descriptor, RANSAC, preserve feature points and feature point homogenization.

Dataset	MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203	ave.	std.
	Easy	Easy	M id	Hard	Hard	Easy	Mid	Hard	Easy	Mid	Hard
a	0.074	0.068	0.065	0.128	0.125	3.333	0.063	0.145	0.048	0.088	0.137	0.535	0.976
b	0.089	0.064	0.092	0.114	0.130	0.052	0.051	0.094	0.055	0.073	0.095	0.021	0.026
c	0.075	0.072	0.066	0.110	0.105	0.052	0.042	0.116	0.057	0.076	0.126	0.023	0.028
d	0.078	0.065	0.065	0.108	0.109	0.057	0.048	0.101	0.055	0.068	0.103	0.019	0.023
e	0.080	0.053	0.071	0.099	0.102	0.063	0.054	0.077	0.063	0.053	0.087	0.014	0.017

.

Table 1 shows that when only the optical flow module is used, the average and standard deviation of position error are the worst, with values of 0.2 and 0.5, respectively. Although the positioning error of combining the module of optical flow tracking, BRIEF descriptor, RANSAC, preserve feature points and feature point homogenization is not the lowest in easy scenarios, its performance is the best in hard scenarios. As the module increases, the average and standard deviation of position error decreases significantly. When combining the module of optical flow tracking, BRIEF descriptor, RANSAC, preserve feature points and feature point homogenization, the average error and standard deviation of position error are 0.014 and 0.017, respectively.

3.2. Compared with Other Methods

We compare the performance of our proposed methods with RANSAC and reverse optical flow method from aspects positioning error, running times, and the number of matching feature points.

3.2.1. Positioning Error

The positioning error results of various feature point mismatch removal methods are shown in

Table 2. Location error (RMSE/m) of various methods on EuRoC dataset, ave. and std. are the average and the standard deviation of location error: a is RANSAC method; b is reverse optical flow method; c from reference [32]; d from reference [33]; e is our proposed method.

Dataset	MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203	ave.	std.
	Easy	Easy	M id	Hard	Hard	Easy	Mid	Hard	Easy	Mid	Hard
a	0.070	0.074	0.067	0.125	0.123	0.052	0.063	0.142	0.058	0.085	0.161	0.093	0.036
b	0.074	0.054	0.082	0.113	0.117	0.061	0.046	0.086	0.056	0.076	0.130	0.081	0.027
c	0.253	-	-	0.380	0.293	-	0.110	0.089	0.166	-	-	0.225	0.123
d	0.137	0.152	0.184	0.276	-	0.112	0.134	0.183	-	-	-	0.168	0.052
e	0.080	0.053	0.071	0.099	0.102	0.063	0.054	0.077	0.063	0.053	0.087	0.014	0.017

. From the experimental results, it can be seen that in easy scenarios such as $MH01-MH03$, $V101-V102$, and $V201-V202$, the RANSAC method can effectively eliminate mismatched points and achieve relatively high positioning accuracy. However, in scenes such as $MH04-MH05$ in a dark environment, $V103$ with varying illumination, and $V203$ with weak texture and poor image quality, RANSAC performs poorly, with an accuracy of only 0.125 m for $MH04-MH05$, and only 0.161 m for $V203$. Our proposed method further improves the accuracy under the influence of illumination changes and image blurring by combining the RANSAC method. The reverse optical flow exhibits an upper middle level in various scenes. Compared to the RANSAC method used by VINS Mono, our proposed method has improved the positioning effect in various scenarios, especially in the light changes of $V103$ and fuzzy weak texture scenarios of $V203$. It is indicating that the method in this paper has better accuracy and robustness for feature point mismatch removal.

3.2.2. Running Times

In addition to accurate feature point matching and positioning accuracy, running speed is also important for visual SLAM. This experiment evaluated the average processing time required for each frame by various methods in 11 sequence datasets of EuRoC. The experimental results are shown in

Table 3. The running time of different method (ms).

Dataset	MH01	MH02	MH03	MH04	MH05	V101	V102	V103	V201	V202	V203
	Easy	Easy	M id	Hard	Hard	Easy	Mid	Hard	Easy	Mid	Hard
RANSAC	0.51	0.52	0.51	0.53	0.53	0.52	0.91	2.19	0.52	1.16	4.31
Reverse optical flow	6.53	6.61	6.82	6.48	6.41	6.50	6.37	6.32	6.39	6.34	6.88
Our method	4.21	4.15	4.42	4.22	4.14	4.06	4.09	4.96	4.14	4.53	6.36

. As can be seen, the processing time of the RANSAC method is relatively short, requiring only about 0.52 ms in an easy Machine Hall scenario. However, as the difficulty of the scene increases, such as factors including lighting changes and image blurring, the time required for RANSAC also increases. In $V203$, the average processing time per frame requires 4.3 ms, which is 8 times that of a normal scene. This is because the quality of feature points in these scenarios is poor, and RANSAC requires multiple iterations to achieve better results. The processing time of reverse optical flow is relatively long, requiring about 6.5 ms. The processing time of our proposed method is in the middle, mainly composed of the total time required for RANSAC and reverse optical flow method.

3.2.3. The Number of Matching Feature Points

In order to verify that the algorithm in this article can retain more accurate matching points, this experiment tested the number of feature point matchings for various algorithms under easy scenarios ($V201$) and hard scenarios ($V203$), as shown in

Table 4. Average matching feature points of different method (total is 260).

Dataset	RANSAC	Reverse Optical Flow	Our Method
V201	228	230	240
V203	134	130	164

. In easy scenarios with less interference, each algorithm can match more feature points, and tracking 260 feature points can match 220 to 240. However, in hard scenes with blurring caused by dithering and large weak texture regions ($V203$), the number of feature points matched by various algorithms decreases sharply, and RANSAC only matches 134 feature points, our proposed method can still match 164 feature points. Figure 3 shows the feature points that RANSAC and our proposed method. As described in Section 2.4, RANSAC removes some correctly matched points, which has little impact in scenarios with more tracking feature points such as $V201$. However, this can have a significant negative impact when the image is blurred, the quality is not high, and the number of feature points tracked is small. As shows in Figure 3a, the number of feature points matched to some images is even less than 20, which can lead to reduced constraints between images, thereby affecting the accurate estimation of posture. Figure 3b shows that our proposed method matches more feature points than RANSAC, ensuring sufficient feature points to provide pose constraints even when the image quality is poor.

4. Conclusions

In this paper, we proposed a multi-constraint feature point tracking method to tackle issues regarding the inability of the single mismatch removal method to adapt to various scenarios. This method eliminates erroneous feature matching pairs extracted by optical flow using multiple constraints such as motion position, gray gradient (descriptor), and geometric constraints (RANSAC), while retaining as many feature points as possible. For the homogenization of feature points, we proposed an image mask homogenization method by dynamically adjusting the radius of the mask. The improved method can effectively balance the quality and uniform distribution of feature points, especially when processing images of poor quality with blurring and weak texture. Finally, the effectiveness of various algorithms and proposed algorithms is tested and compared on the EuRoC dataset, and the results show that our proposed method is robust and effective in easy scenes and difficult scenarios such as those containing light changes, image blurring, and weak textures.

In future work, we hope to integrate deep learning to more effectively extract high-quality feature points. Alternatively, we can directly utilize the features output by deep learning, rather than being limited to feature points.