Vision-Based Moving-Target Geolocation Using Dual Unmanned Aerial Vehicles

Abstract

This paper develops a framework for geolocating ground-based moving targets with images taken from dual unmanned aerial vehicles (UAVs). Unlike the usual moving-target geolocation methods that rely heavily on accurate navigation state sensors or assumptions of the known target’s altitude, the proposed framework does not have the same limitations and performs geolocation of moving targets utilizing dual UAVs equipped with the low-quality navigation state sensors. Considering the Gaussian measurement errors and yaw-angle measurement bias provided by low-quality sensors, we first propose an epipolar constraint-based corresponding-point-matching method, which enables the historical measurement data to be used to estimate the current position of the moving target; after that, we propose a target altitude estimation method based on multiview geometry, which utilizes multiple images, including historical images, to estimate the altitude of the moving target; finally, considering the negative influence of yaw-angle measurement bias on the processes of target altitude estimation and parameter regression, we take advantage of multiple iterations among the two processes to accurately estimate the moving target’s two-dimensional position and the yaw-angle measurement biases of two UAVs. The effectiveness and practicability of the framework proposed in this paper are proved by simulation experiments and actual flight experiments.

1. Introduction

In recent years, with the advantages of small size, simple operation, and low price [1,2,3], unmanned aerial vehicles (UAVs) are increasingly being deployed for a wide variety of missions, including surveillance and reconnaissance. UAV-based computer vision capabilities, such as target detection and tracking, play particularly important roles in the above missions. However, it is not sufficient to simply detect and track a moving target; we often need to know the three-dimensional position of the moving target, which is obtained by the process of target geolocation [4,5,6]. In this paper, target geolocation refers to the process of using UAVs to obtain target information and estimate the position of the target in the world coordinate system. In this process, GPS (global positioning system) and AHRS (attitude and heading reference system) are used to acquire the UAV’s navigation state, such as position and attitude, for geolocation in real time; the UAV’s navigation state and the target’s image are acquired simultaneously. There have been many studies on ground target geolocation technology [7,8,9]. Fabian et al. [10] analyzed the factors that affect the accuracy of moving-target geolocation. Generally speaking, a precise geolocation result depends on accurate navigation state measurement of UAVs and target’s altitude. Therefore, it is difficult to obtain the accurate geolocation result of the non-cooperative target utilizing the onboard low-quality navigation state sensors. For this reason, this paper aims to achieve accurate geolocation by using only vision to mitigate the effects of Gaussian measurement errors and the yaw-angle measurement bias, and to accurately estimate the position of the grounded moving target, which is a challenge.

In recent years, scholars have conducted a considerable amount of research on moving-target geolocation using UAV systems. In general, moving-target geolocation methods can be divided into two categories: the methods based on one-shot learning [11,12,13,14] and the methods based on multiview geometry [15,16,17]. As the most commonly used method, the one-shot method only obtains one image of the moving target but require the altitude of the moving target. There are many ways to get the altitude information of the moving target. Some researchers utilized a DEM (digital elevation model) to obtain the altitude of the moving target. Qiao et al. [18] proposed a vision-based geolocation frame could estimate the three-dimensional position of a moving target in real time with a gimbal-stabilized camera, but an accurate digital elevation model is required to obtain the altitude of a moving target on the ground. Alternatively, some researchers obtained the target’s altitude through estimating the distance between the UAV and the moving target based on the size of the target in the image. Zhang et al. [19] proposed a relative distance estimation method based on the prior information of the target that required the size of the target to be known and then used the geometric relationship to calculate the distance between the UAV and the target. Zhu et al. [20] showed that their learning-based method could estimate the distance to a specific target. Nevertheless, their experimental results showed that the distance estimation methods based on prior information work well only if the target is close and are not suitable for accurately geolocating grounded targets with UAVs. In practical applications, the distance from the UAV to the target is usually determined by a laser rangefinder [11], which has the highest accuracy among the above-mentioned methods. However, the use of laser-based methods for the continuous geolocation of moving targets will greatly reduce the flight time of UAVs due to the weight of the laser rangefinder. It is worth noting that the above-mentioned methods all assume that the UAV’s navigation state is accurate; that is, they do not consider the negative influence of the measurement errors of the UAV’s navigation state on the ground moving-target geolocation.

To avoid the limits of the above-mentioned methods, some researchers utilized multiple UAVs to simultaneously acquire multiple images of a moving target and estimated its position by multiview geometry. Wang et al. [21] utilized multiple UAVs to geolocate a moving target and presented nonlinear filter based on solving the Fokker–Planck equation to address the issue of the time delay during data transmission. Xu et al. [22] proposed a method to adaptively adjust the weights according to the positions of the UAVs, which can improve the accuracy of the results of the weighted least-squares. However, these methods did not solve the problems well. On the one hand, mitigating the negative influence of Gaussian measurement errors on geolocation requires multiple images, and each UAV can only obtain one image at a time, which means that a large number of UAVs are required for accurate moving-target geolocation. On the other hand, although the above-mentioned methods using multiple measurements can mitigate the influence of Gaussian measurement errors provided by the navigation state sensors, they still have not solved the problem of yaw-angle measurement bias.

The yaw-angle bias provided by the AHRS may vary every time when the AHRS is initialized, and the assumption of unknown and constant bias is justified in a short time window [23]. Related to this, Zhang et al. [24] proposed a vision-based target geolocation method, which focuses on eliminating the effects of yaw-angle measurement bias provided by low-quality sensors on target geolocation. Further, Zhang et al. [25] focused on improving the estimation accuracy of yaw-angle bias by utilizing a particle swarm optimization algorithm to derive an expression of the optimal trajectory and perform optimal trajectory planning and then to enhance the geolocation performance. However, their methods can only be applied to the stationary-target geolocation scenarios.

In the scenario assumed in this paper, two UAVs follow a single moving target on the ground and estimate the position of the moving target in real time. Except for the images of the moving target, all the information of the target is unknown. Both UAVs are equipped with monocular cameras to obtain target images. In addition, UAVs are equipped with GPS and AHRS to obtain their own positional and attitude information. In the navigation state measurement of UAVs, there are Gaussian measurement errors for position and attitude information, and due to different measurement principles, there are also measurement biases for yaw angle. The purpose of this paper is to mitigate the negative impact of UAV navigation state measurement errors on moving-target geolocation accuracy in the above scenario. In order to deal with this issue, we develop a ground-based moving-target geolocation framework utilizing dual UAVs, and the constructed system is shown in Figure 1. The proposed framework is free from the above-mentioned constraint of assuming the ground target is stationary, and it geolocates the moving target using sequential images taken from dual UAVs and navigation states provided by low-quality sensors.

The main contributions of this paper are as follows.

• We propose a ground moving-target geolocation framework that utilizes only the sequence of images obtained by two UAVs with low-quality sensors. The framework contains the processes of corresponding point matching, target altitude estimation, and parameter regression, which are used to mitigate the negative influences of Gaussian measurement errors and yaw-angle biases.
• In order to obtained more available images using only two UAVs, we propose a corresponding-point-matching method based on epipolar constraint. The method enables the historical images to be used to estimate the current position of the moving target. In addition, in order to filter out the wrong corresponding points, we propose an outer point filtering method based on the consistency principle.
• The effectiveness of the proposed framework was verified via the experiments in simulated and real environments.

2. Methods

The proposed ground moving-target geolocation framework based on dual UAVs is illustrated in Figure 2. The framework utilizes a multiprocess architecture to process the four tasks of data acquisition, target detection, corresponding point matching, and moving-target geolocation in parallel. Moreover, in the moving-target geolocation, the two subtasks of target altitude estimation and parameter regression are processed iteratively.

The data acquisition process uses two UAVs equipped with a camera and low-quality navigation sensors to track the moving target and obtain the images of the target and the navigation states of two UAVs simultaneously; then, the images and navigation states are transmitted to the ground station in real time. The target detection process obtains the latest image from each UAV and achieves accurate detection of the moving target with the image pair. The corresponding point matching is to provide sufficient measurement data for the subsequent moving-target geolocation; it matches the corresponding points in the past images according to the target points provided by the target detection process, and then the past frames’ images can be utilized to estimate the current position of the moving target. The process of target altitude estimation aims to estimate the target’s altitude utilizing the current images and the past images. After that, the target’s altitude is used by the parameter regression process to estimate the two-dimensional position of the target and the yaw-angle measurement biases of two UAVs. Considering the yaw-angle measurement biases, there are several iterations between the target altitude estimation and the parameter regression until the estimated yaw-angle biases are less than the given threshold or the number of iterations reaches the given maximum.

2.1. Parameter Regression

The parameter regression is used to estimate the two-dimensional position of target and the yaw-angle measurement biases of two UAVs. The target geolocation model based on UAV can be visualized in Figure 3.

According to [24], it can be expressed as

$$\left[\begin{array}{c}{x}_P\\ y_P\end{array}\right]=\left[\begin{array}{c}{x}_v\\ y_v\end{array}\right]+\frac{\left(z_P-z_v\right)}{\left(0,0,1\right)C_w^c\left[\begin{array}{c}{x}_p\\ y_p\\ f\end{array}\right]}\times \left[\begin{array}{c}\begin{array}{ccc}1& 0& 0\end{array}\\ \begin{array}{ccc}0& 1& 0\end{array}\end{array}\right]C_w^c\left[\begin{array}{c}{x}_p\\ y_p\\ f\end{array}\right]$$

(1)

where ${(x_P,y_P,z_P)}^T$ and ${(x_v,y_v,z_v)}^T$ represent the coordinates of the current target’s position P and the UAV’s position in the world coordinate system w, respectively. The coordinates of projection point p of current target’s position P are represented as $(x_p,y_p)$ in the image plane; and f is the focal length of the camera. In a simulation experiment, the precise focal lengths can be obtained directly. In a real experiment, we used the kit provided by ROS [26] to calibrate the cameras to obtain the focal lengths. The UAV’s navigation state $(x_v,y_v,z_v,\psi ,\theta ,\varphi )$ was measured by the onboard GPS and AHRS. The rotation matrix $C_c^w$ represents the transformation from the camera coordinate system c to the world coordinate system w. The relative altitude h between the current target’s position and the UAV is defined as

$$h=z_v-z_P$$

(2)

In general, the pitch angle $\theta$ and the roll angle $\varphi$ of a UAV can be measured accurately, but the yaw-angle measurement will have large error due to the use of magnetometer. Hence, to solve the above-mentioned issue, we estimate the yaw-angle measurement biases of two UAVs and target’s position by parameter regression. The estimation process is illustrated as follows.

The goal of parameter regression is to estimate the parameter $\xi ={[{\xi }_1,{\xi }_2]}^T$, where

$${\xi }_1:={[x_P,y_P]}^T$$

(3)

is the two-dimensional coordinate of the moving target and

$${\xi }_2:={[\delta {\psi }_L,\delta {\psi }_R]}^T$$

(4)

are the yaw-angle measurement biases of the left UAV and right UAV, respectively. The measured navigation state of two UAVs is $\tau ={[{\tau }_1,{\tau }_2,{\tau }_3]}^T$, where

$${\tau }_1:={[x_{v_L},y_{v_L},z_{v_L},x_{p_L},y_{p_L},{\theta }_L,{\varphi }_L]}^T$$

(5)

is the navigation state of the left UAV and

$${\tau }_2:={[x_{v_R},y_{v_R},z_{v_R},x_{p_R},y_{p_R},{\theta }_R,{\varphi }_R]}^T$$

(6)

is the navigation state of the right UAV. The measured navigation state ${\tau }_3:={[{\psi }_L,{\psi }_R]}^T$ includes the yaw angles of the left UAV and the right UAV, respectively. According to (1), the general form of the measurement equation is as follows:

$${\xi }_1=F({\tau }_1,{\tau }_2,{\tau }_3)$$

(7)

and the actual measurements are

$${\gamma }_1={\tau }_1+w_1,w_1\sim N(0,Q_1)$$

(8)

$${\gamma }_2={\tau }_2+w_2,w_2\sim N(0,Q_2)$$

(9)

$${\gamma }_3={\tau }_3+{\xi }_2+w_3,w_3\sim N(0,Q_3)$$

(10)

where the Gaussian measurement noise covariances $Q_1$ and $Q_2$ are both $7\times 7$ symmetric positive-definite matrices, and $Q_3$ is a $2\times 2$ symmetric positive-definite matrix. Combine (8), (9) and (10) into the measurement equation as follows:

$${\xi }_1=F({\gamma }_1-w_1,{\gamma }_2-w_2,{\gamma }_3-({\xi }_2+w_3))$$

(11)

Then, the linearized measurement equation is obtained by using Taylor’s theorem:

$$\begin{array}{cc}\hfill F({\gamma }_1,{\gamma }_2,{\gamma }_3)\approx {\xi }_1& +{\left.\frac{\partial F}{\partial {\tau }_1}\right|}_{{\gamma }_1,{\gamma }_2,{\gamma }_3}·w_1+{\left.\frac{\partial F}{\partial {\tau }_2}\right|}_{{\gamma }_1,{\gamma }_2,{\gamma }_3}·w_2+{\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_1,{\gamma }_2,{\gamma }_3}·w_3\hfill \\ & +{\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_1,{\gamma }_2,{\gamma }_3}·{\xi }_2\hfill \end{array}$$

(12)

Suppose that $2N$ bearing measurements of the moving target are taken by the left UAV and the right UAV at the discrete time $t=1,\dots ,N$; that is, $({{\gamma }_1}_1,{{\gamma }_2}_1,{{\gamma }_3}_1)$,…,$({{\gamma }_1}_N,{{\gamma }_2}_N,{{\gamma }_3}_N)$. For $2N$ bearing measurements, the linear measurement equations are as follows:

$$\left(\begin{array}{c}F({\gamma }_{1_1},{\gamma }_{2_1},{\gamma }_{3_1})\\ ·\\ ·\\ F({\gamma }_{1_N},{\gamma }_{2_N},{\gamma }_{3_N})\end{array}\right)=\left[\begin{array}{cc}{I}_2& {\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_{1_1},{\gamma }_{2_1},{\gamma }_{3_1}}\\ ·& ·\\ ·& ·\\ I_2& {\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_{1N},{\gamma }_{2_N},{\gamma }_{3_N}}\end{array}\right]\xi +W$$

(13)

where

$$W\sim N(0,Q)$$

(14)

is the geolocation error caused by the Gaussian measurement error of the UAV’s navigation state, and its convariance matrix is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}Q=diag\left(\left\{\left({\left.\frac{\partial F}{\partial {\tau }_1}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)Q_1{\left({\left.\frac{\partial F}{\partial {\tau }_1}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)}^T\right.\right.+\left({\left.\frac{\partial F}{\partial {\tau }_2}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)Q_2{\left({\left.\frac{\partial F}{\partial {\tau }_2}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)}^T\hfill \\ +\left.{\left.\left({\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)Q_3{\left({\left.\frac{\partial F}{\partial {\tau }_3}\right|}_{{\gamma }_{1_k},{\gamma }_{2_k},{\gamma }_{3_k}}\right)}^T\right\}}_{k=1}^N\right)\hfill \end{array}\end{array}$$

(15)

From (13), the two-dimensional position of moving target and the yaw-angle biases of two UAVs can be solved effectively with the weighted least-square method.

2.2. Estimation of Target Altitude

It can be seen in (1) that the target’s altitude $z_P$ is required by the process of parameter regression. In this section, the target’s altitude $z_P$ is obtained by utilizing the multiview geometry.

The left UAV and the right UAV continuously acquire the navigation state and the target images when tracking the moving target. We select the current image and $N-1$ past images from the sequence of images taken by each UAV as the measurement data to estimate the current position of the moving target, and the two adjacent images in these N images must meet the baseline constraint—that is, the distance between the positions of UAV when taking these two images must be greater than or equal to the given threshold. Any image ${g_L}_i(i=1,...,N)$ taken by the left UAV and any image ${g_R}_j(j=1,...,N)$ taken by the right UAV can be form an image pair $m_{ij}$, and a total of $N^2$ image pairs can be obtained. For each image pair $m_{ij}$, its multiview geometry model is shown in Figure 4. The points $p_L$ and $p_R$ are the corresponding points of the target’s position P projected in the left image, ${g_L}_i$, and the right image, ${g_R}_j$, respectively. The corresponding points in the current images are provided by target detection process, but the corresponding points in the past images are provided by the process of corresponding point matching described in Section 2.3. $O_L$ and $O_R$ are the origins of the left and right camera coordinate systems, respectively. According to the principle of multiview geometry, if there is no measurement error, the lines $O_L{p}_L$ and $O_R{p}_R$ will intersect at point P. Nevertheless, due to the Gaussian measurement error and yaw-angle biases provided by the low-quality sensors, the lines will not intersect at the point P. Therefore, we approximate the intersection point P to the nearest point $P^{{}^{\prime }}$ from the two lines.

Let $a{p}_L(a\in R)$, $T_{RL}+b{R}_{RL}^T{p}_R(b\in R)$ be ray $O_L{p}_L$ and ray $O_R{p}_R$ in the left coordinate system, respectively. We denote J as an orthogonal vector with the rays $O_L{p}_L$ and $O_R{p}_R$. The problem is now simplified to searching the midpoint $P^{{}^{\prime }}$ of the line segment parallel to J that joins $O_L{p}_L$ and $O_R{p}_R$.

The endpoints of the line segment ($a_0{p}_L$ and $T_{RL}+b_0{R}_{RL}^T{p}_R$) are calculated to solve the following equation:

$$\left[\begin{array}{ccc}{p}_L& -R_{RL}^T{p}_R& p_L\times \end{array}{R}_{RL}^T{p}_R\right]\left[\begin{array}{c}a\\ b\\ c\end{array}\right]=T_{RL}$$

(16)

for $a_0$, $b_0$, and $c_0$. We let $h_{ij}$ be the relative altitude between the target and the UAV when taking the image ${g_L}_i$, and $h_{ij}$ is estimated utilized image ${g_L}_i$ and ${g_R}_j$. Similarly, $h_{ji}$ is the relative altitude between the target and the UAV when taking the image ${g_R}_j$, and $h_{ji}$ is estimated utilizing image ${g_R}_j$ and ${g_L}_i$. Thus, the relative altitude $h_{ij}$ and $h_{ji}$ can be obtained from the following equations:

$$h_{ij}=\frac{1}{2}\left[\begin{array}{ccc}0& 0& 1\end{array}\right]R_L\left(a_0{p}_L+b_0{R}_{RL}{p}_R+T_{RL}\right)$$

(17)

$$h_{ji}=\frac{1}{2}\left[\begin{array}{ccc}0& 0& 1\end{array}\right]R_R\left(b_0{p}_R+a_0{R}_{LR}{p}_L+T_{LR}\right)$$

(18)

where $p_L:={[{x_p}_L,{y_p}_L,f_L]}^T$ and $p_R:={[{x_p}_R,{y_p}_R,f_R]}^T$ are the coordinates of the corresponding points in the left and right coordinate systems, respectively. The matrices $R_{RL}$ and $T_{RL}$ are the rotation matrix and displacement matrix from the right coordinate system to the left coordinate system, respectively. Similarly, the matrices $R_{LR}$ and $T_{LR}$ are the rotation matrix and displacement matrix from the left coordinate system to the right coordinate system respectively. That is,

$$R_L:=C_c^n({\psi }_i+\delta {{\psi }_L}_n,{\theta }_i,{\varphi }_i),R_R:=C_c^n({\psi }_j+\delta {{\psi }_R}_n,{\theta }_j,{\varphi }_j)$$

(19)

where $\delta {{\psi }_L}_n$ and $\delta {{\psi }_R}_n$ is the estimated value given by parameter regression in the $n_{th}$ iteration and the initial values $\delta {{\psi }_L}_0=\delta {{\psi }_R}_0=0$. There are N image pairs containing image ${g_L}_i$ or image ${g_R}_j$, so we can get N estimates of relative altitude for each image ${g_L}_i$ or image ${g_R}_j$. Thus, we can get the average altitude ${\overline{h}}_i$ and ${\overline{h}}_j$ for image ${g_L}_i$ and ${g_R}_j$ by the following equations, respectively.

$${\overline{h}}_i=\frac{{\displaystyle \sum _{j=1}^N}{h}_{ij}}{N},{\overline{h}}_j=\frac{{\displaystyle \sum _{i=1}^N}{h}_{ji}}{N}$$

(20)

After that, we estimate the target’s altitude according to the indirect adjustment theory. After the estimation of relative altitude, each image $g_k(k=1,...,2N)$ has an estimated relative altitude ${\overline{h}}_k$. There is estimation error for relative altitude ${\overline{h}}_k$ and measurement error for UAV’s altitude $z_{v_k}$. According to (2), we have

$$z_{v_k}+e_{v_k}=z_P+{\overline{h}}_k+e_{h_k}$$

(21)

where $e_{v_k}$ is the measurement error of $z_{v_k}$ and $e_{h_k}$ is the estimation error of ${\overline{h}}_k$. For $2N$ images, we use $E_k$ to represent $e_{v_k}-e_{h_k}$ and have

$$\left(\begin{array}{c}{E}_1\\ ·\\ ·\\ E_{2N}\end{array}\right)=J_{2N,1}{z}_P-\left[\begin{array}{c}{z}_{v_1}-{\overline{h}}_1\\ ·\\ ·\\ z_{v_{2N}}-{\overline{h}}_{2N}\end{array}\right]$$

(22)

where $J_{2N,1}$ represents the $2N\times 1$ matrix of ones. According to the least-squares theory, $z_P$ can be obtained as follows:

$$z_P={\left(J_{2N,1}^T{J}_{2N,1}\right)}^{-1}{J}_{2N,1}^T\left[\begin{array}{c}{z}_{v_1}-{\overline{h}}_1\\ ·\\ ·\\ z_{v_{2N}}-{\overline{h}}_{2N}\end{array}\right]$$

(23)

Finally, the estimated altitude of the moving target is used to estimate the yaw-angle biases and the two-dimensional positions described in Section 2.1.

2.3. Corresponding Point Matching

In this paper, the corresponding points refer to the projection points of the target’s current position P on different images. The target points provided by target detection in the current images ${g_L}_1$ and ${g_R}_1$ captured by the left and right UAVs, respectively, are corresponding points of the moving target’s current position, but the target points provided by target detection in the past images cannot be regarded as corresponding points because the target’s position has changed. Therefore, it is necessary to match the corresponding points before using the past images to estimate the current position of the moving target.

In this section, we first propose a corresponding-point-matching method for the current images and the past images based on an epipolar constraint. As mentioned above, $2N$ images are required for each estimation of target position, of which N images are from the left camera and the other N images are from the right camera. Among the $2N$ images, there are two current images, denoted as ${g_L}_1$ and ${g_R}_1$, and the remaining $2N-2$ images are the past images. For each matching of the corresponding point, the current images ${g_L}_1$ and ${g_R}_1$ and any past image $g_n(n=1,...,2N-2)$ are used to build a corresponding-point-matching model, as shown in Figure 5. The points $p_L:={[{x_p}_L,{y_p}_L]}^T$ and $p_R:={[{x_p}_R,{y_p}_R]}^T$ are the corresponding points provided by target detection in the current images ${g_L}_1$ and ${g_R}_1$ captured by the left UAV and right UAV. The point $p_n:={[x_p,y_p]}^T$ is the corresponding point provided by the corresponding point matching in the past image $g_n$. $O_L$, $O_R$, and $O_n$ are the optical centers of cameras. The points $e_1$, $e_L$, and $e_2$, $e_R$ are the intersection points of the lines $O_L{O}_n$ and $O_R{O}_n$ with the image plane, respectively.

The coordinate of target point $p_L$ in the camera coordinate system $O_n$ is represented as

$$p_L^{\prime }=R_{Ln}\left(K_L^{-1}{\left[x_{p_L},y_{p_L},1\right]}^T\right)+T_{Ln}$$

(24)

where the matrix $K_L$ is the intrinsic matrix of the left camera. The matrices $R_{Ln}$ and $T_{Ln}$ are the rotation matrix and translation matrix from coordinate system $O_L$ to coordinate system $O_n$, respectively. The coordinate of target point $p_R$ in the camera coordinate system $O_n$ is represented as

$$p_R^{\prime }=R_{Rn}\left(K_R^{-1}{\left[x_{p_R},y_{p_R},1\right]}^T\right)+T_{Rn}$$

(25)

where the matrix $K_R$ is the intrinsic matrix of the right camera. The intrinsic matrix K reflects the attributes of the camera, and its description is as follows:

$$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$$

(26)

where $f_x$ represents the ratio of the physical focal length to the physical size of the pixel in the x direction, and $f_y$ represents the ratio of the physical focal length to the physical size of the pixel in the y direction. For the cameras used in simulation experiments and real experiments, there is $f_x=f_y=f$. The coordinate of the principal point of photograph is denoted as $(c_x,c_y)$. The intrinsic matrix K is obtained by camera calibration, and the intrinsic matrices of the left camera and the right camera are denoted as $K_L$ and $K_R$, respectively. The matrices $R_{Rn}$ and $T_{Rn}$ are the rotation matrix and translation matrix from coordinate system $O_R$ to coordinate system $O_n$, respectively. In the corresponding-point-matching model, the three-dimensional point P may be at any position on the line $O_{L}P$ because its depth information is unknown. However, no matter where the three-dimensional point P is on the line $O_{L}P$, its projection point $p_n$ on image $g_n$ must be on the line $e_1{p}_n$ according to the imaging principle. The line $e_1{p}_n$ can be described as

$$\left[T_{Ln}\times {p^{\prime }}_L\right]K_n^{-1}{\left[x_p,y_p,1\right]}^T=0$$

(27)

where $K_n^{-1}{\left[x_p,y_p,1\right]}^T$ is the coordinate of point $p_n$ in the camera coordinate system $O_n$. Similarly, the line $e_2{p}_n$ can also be determined on the image $g_n$ according to the target point $p_R$, which can be described as

$$\left[T_{Rn}\times {p^{\prime }}_R\right]K_n^{-1}{\left[x_p,y_p,1\right]}^T=0$$

(28)

Therefore, the coordinates of corresponding point $p_n$ on the image $g_n$ can be solved by combining (27) and (28). In other words, the corresponding point we are looking for is the intersection of line $e_1{p}_n$ and line $e_2{p}_n$. In the above process, the rotation and translation matrices between cameras are necessary. For this, the corresponding point matching subtask first extracts ORB features from the current images, which is an effective and fast-computing feature point. Then, the subtask matches the feature points extracted from these two current images with the feature points extracted from the past image. After that, the eight-point algorithm with random sample consensus (RANSAC) is used to improve the accuracy of feature point matching and estimate the fundamental matrices $F_{Ln}$ and $F_{Rn}$. The rotation and translation matrices between cameras can be obtained by decomposing the fundamental matrices $F_{Ln}$ and $F_{Rn}$.

However, the error matching of the feature points extracted from the current images and the past image will reduce the accuracy of the fundamental matrix estimation and then reduce the accuracy of the corresponding point matching. In this regard, we propose a method based on the consistency principle to filter the wrong corresponding points. The subtask of corresponding point matching can obtain $2N-2$ corresponding points because there are $2N-2$ past images selected to be used to estimate the current position of the moving target. Then, the two current images and each of the $2N-2$ past image form multiple groups of observation data—that is, $({g_L}_1,{g_R}_1,g_1)$,…,$({g_L}_1,{g_R}_1,g_{2N-2})$. Each group is sent to the iterations of target altitude estimation and parameter regression. After that, we can get $2N-2$ estimates of yaw-angle biases—that is, ${\xi }_{2_1}$,…, ${\xi }_{2_{2N-2}}$. For each estimate ${\xi }_{2_n}(n=1,...,2N-2)$, we calculate its consistency factor $C_n$ according to the following equation:

$$C_n=\frac{{\displaystyle \sum _{m=1;m\ne \mathrm{n}}^{2N-2}}\mathbb{I}\left(e_{(n,m)}\right)}{2N-2}$$

(29)

$\mathbb{I}(\cdot )$ is an indicator function defined by

$$\mathbb{I}\left(e_{(n,m)}\right)=\left\{\begin{array}{cc}1,\hfill & if{e}_{(n,m)}<TH\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.$$

(30)

where $e_{nm}$ is defined by

$$e_{nm}=abs(\delta {\psi }_L^n-\delta {\psi }_L^m)+abs(\delta {\psi }_R^n-\delta {\psi }_R^m)$$

(31)

After that, we have $C_{max}=MAX(C_1,...,C_{2N-2})$ and its corresponding estimates ${\xi }_{2_n}={[\delta {\psi }_L^n,\delta {\psi }_R^n]}^T$. The set $S_p$ of corresponding points to be filtered is obtained; that is,

$$S_p=\left\{p_m|e_{nm}\ge TH\right\}$$

(32)

The corresponding points $p_m$ in set $S_p$ are considered as incorrect corresponding points, and their corresponding image $g_m$ will no longer be used to estimate the position for the moving target. It should be noted that if $N=2$ and $e_{nm}\ge TH$, the two past images are eliminated and the geolocation is terminated because there is not enough measurement data.

In this section, a corresponding-point-matching method based on epipolar constraint is proposed. Considering the existence of a wrong corresponding point, we also propose a method based on the consistency principle to eliminate the wrong corresponding points. After corresponding point matching, multiple past images can be used to estimate the current position of the moving target.

3. Results

In this section, we evaluate the performance of the proposed geolocation framework for a moving target. In order to conduct the simulation experiments, we built a moving-target geolocation platform based on ROS and GAZEBO [26] to evaluate the robustness of the proposed framework by various simulations. In addition, we used two quad-rotor UAVs to further verify the effectiveness of the framework in a real experiment.

3.1. Simulation Experiment

For simulation experiments, a moving target detection and tracking platform in an urban environment was built. In the platform, a vehicle runs in a city environment, as shown in Figure 6, and two UAVs with vertically downward cameras detect the target utilizing Yolov5 [27], as shown in Figure 7. The flight altitude of the two UAVs is 100 m, and the baseline between the two UAVs is 70 m. In the process of target sensing, the UAVs first use the target detection algorithm to find the target, and then use the proposed method to continuously geolocate the moving target. We implemented the proposed framework using Python and ran the framework on a machine with Intel(R) an i7-10700 @2.90 GHz CPU and NVIDIA RTX 2070 Super GPU.

Before geolocating the moving target using multiple images, including the past images, the corresponding points in the past images were matched using the corresponding-point-matching method described in Section 2.3. The accuracy of the corresponding-point-matching method proposed in this paper depends heavily on the estimation of the fundamental matrix F, and wrong corresponding point matching will cause an error in target geolocation. However, the estimation of the fundamental matrix is occasionally inaccurate due to the matching error of ORB features between the current image and the past image. Therefore, the method of filtering the wrong corresponding points is proposed in Section 2.3.

In order to evaluate the performance of the method for corresponding point matching and the effectiveness of the method for filtering the wrong corresponding points, we obtained the experimental results shown in Figure 8. In this experiment, we did not consider the influence of UAV navigation state measurement error on the geolocation results, but only observed the changes caused by the corresponding points. It is worth noting that eight images were used for each geolocation in this experiment. In Figure 8, the black line is the ground-truth path of the moving target, and the yellow line is the path estimated by the proposed moving-target geolocation framework without filtering the wrong corresponding points. It can be seen that the corresponding point matching introduces errors to the moving-target geolocation, and the geolocation error is large when the car is at the corner of an image, which is due to the small overlap between the current image and the past images, so there are not enough ORB features to estimate an accurate fundamental matrix. However, the geolocation error was greatly reduced after using the proposed method of filtering the wrong matching points, as shown by the red line.

The statistical characteristics of continuous geolocation results are shown in

Table 1. Statistical results of geolocation in the experiment of corresponding point matching.

	Before Filtering			After Filtering
	x	y	z	x	y	z
MAE (m)	0.283	0.274	1.216	0.151	0.173	0.720
STD (m)	0.373	0.408	1.561	0.125	0.164	0.547
MAX (m)	4.135	6.335	8.574	0.692	1.135	2.464

, in which we count the mean absolute errors (MAEs), the standard deviations (STDs), and the maximum errors (MAX) of geolocation results in each direction, X, Y, and Z. The mean absolute error (MAE) is calculated as follows:

$$MAE=\frac{{\displaystyle \sum _{i=1}^n}\left|y_i-{\widehat{y}}_i\right|}{n}$$

(33)

where n represents the number of geolocations, and $y_i$ and ${\widehat{y}}_i$ represent the estimated value and the true value, respectively. It can be seen in Table 1 that the proposed method can effectively filter out the wrong corresponding points and reduce the geolocation errors.

After matching corresponding points, the past images and the current images are used to estimate the target’s altitude $z_P$. We obtained the statistical characteristics of altitude estimation in the continuous geolocation process, as shown in

Table 2. Statistical results of altitude estimation of the target.

Number of Images	MAE $\left(\mathit{m}\right)$	STD $\left(\mathit{m}\right)$	MAX $\left(\mathit{m}\right)$
4	3.372	2.769	9.812
8	2.448	1.931	7.819

. We utilized 4 or 8 images to obtain the altitude estimation results, and compared the mean absolute errors, the standard deviations, and the maximum errors of altitude estimation errors when using different numbers of images. In addition, we set ${\sigma }_{\psi }={\sigma }_{\theta }={\sigma }_{\phi }=2^{{}^{\circ }}$ and ${\sigma }_{x_v}={\sigma }_{y_v}={\sigma }_{z_v}=3m$. The yaw-angle biases $\delta {\psi }_L,\delta {\psi }_R$ of the left and right UAVs were set as follows:

$$\delta {\psi }_L=0.2t+\delta {\psi }_0;\delta {\psi }_R=0.2t-\delta {\psi }_0$$

(34)

where $t\left(s\right)$ represents the flight time of UAVs and $\delta {\psi }_0$ was set to 20 in this experiment. It can be seen form the results shown in Table 2 that the more images used for target altitude estimation, the more accurate the altitude estimation will be. However, using more images means more computing costs, and the average processing speed of moving-target geolocation was 22 FPS when utilizing eight images and the above-mentioned machine in the realistic experiment.

Similarly to the experiments of target altitude estimation, we evaluated the performance of yaw-angle bias estimation. In this experiment, the true values of yaw-angle biases need to be known in order to evaluate our method, but it is difficult to measure the yaw-angle biases of a sensor in actual flight. Therefore, we set the same experimental conditions as the altitude estimation experiment to evaluate the performance of yaw-angle biases’ estimation in the simulation environment. As shown in

Table 3. Statistical results of yaw-angle bias estimation.

Number of Images		MAE $\left(\mathit{deg}\right)$	STD $\left(\mathit{deg}\right)$	MAX $\left(\mathit{deg}\right)$
4	$\delta {\psi }_L$	7.642	4.270	13.943
	$\delta {\psi }_R$	6.633	3.477	13.125
8	$\delta {\psi }_L$	3.051	2.781	7.099
	$\delta {\psi }_R$	2.550	2.156	6.957

, the same conclusion can be obtained—that is, the proposed method can effectively estimate the yaw-angle biases of UAVs, and the more images used, the more reliable the estimation results will be.

In the proposed framework, the altitude of a moving target and the yaw-angle biases of two UAVs are estimated iteratively until the preset conditions are reached. We preset two conditions for stopping the iteration. One is that the estimated yaw-angle biases of the current iteration are less than 0.1, and the other is that the maximum number of iterations is reached, and we set the maximum number of iterations to 20 in this study.

Table 4. Iterative process of parameter estimation.

Number of Iterations k	1	2	3	4	8	12	16	20
${\delta {\psi }_L}_k$$\left(deg\right)$	$16.372$	$4.269$	$2.812$	$2.753$	$2.841$	$2.912$	$2.736$	$3.051$
${\delta {\psi }_R}_k$$\left(deg\right)$	$14.448$	$4.931$	$2.469$	$2.817$	$2.526$	$2.728$	$2.837$	$2.550$
$z_P$$\left(m\right)$	10.448	5.931	3.819	2.769	2.815	2.635	2.864	2.448

shows the average absolute errors of the estimated values in the iteration process of the proposed framework utilizing eight images under the above-mentioned experimental settings. It can be seen that when the number of iterations is four, the estimated yaw-angle biases and target altitude reach stable values. After that, the results of subsequent iterations fluctuate around the stable values. This is because in the iteration process, there is not only the influence of yaw-angle biases, but also the influence of Gaussian measurement errors. This makes it impossible for the whole system to reach a minimum yaw-angle biases through multiple iterations.

The main factors that affect the geolocation accuracy are the measurement errors of the UAV’s navigation state. Therefore, we evaluated the performance of the proposed framework under different measurement errors of the UAV’s attitude angle and position. As the comparison algorithm, the one-shot method assumes that the distance from UAV to the moving target is known (e.g., provided by laser rangefinder) and utilizes only one image to estimate the three-dimensional coordinates of the moving target. This approach assumes that the UAV navigation state is provided by an accurate AHRS, so it does not consider the influence of the UAV navigation state measurement errors on the geolocation result. We first assume that only the attitude angle has Gaussian measurement error in the UAV’s navigation state, and then compares the performance of the proposed framework and the one-shot method. Specifically, we set ${\sigma }_{\psi }=1^{\circ },\dots ,5^{\circ }$, respectively, and ${\sigma }_{\psi }={\sigma }_{\theta }={\sigma }_{\phi }$. In the comparisons, we compared the three-dimensional geolocation errors and the distance error, as shown in Figure 9. It should be noted that the distance error is the distance between the estimated three-dimensional position and the true three-dimensional position.

The proposed framework utilizes 4 or 8 images to obtain the geolocation results for each geolocation. It can be seen that with an increase in attitude-angle measurement errors, the geolocation errors obtained by the two methods increase. In terms of the x coordinate and y coordinate, the geolocation errors of our method are always smaller than those of the one-shot method. This is because the proposed framework can expand the measurement data by corresponding point matching, and more measurement data can mitigate the negative influence of Gaussian measurement errors. In the same way, for the proposed framework, the geolocation results obtained by using eight images are more stable than those obtained by using four images. However, in terms of z coordinate, the geolocation error of our method is larger than that of the one-shot method. There are two reasons for this phenomenon. On the one hand, the target’s altitude $z_P$ is calculated according to the distance between the UAV and the moving target, and the true distance is assumed to be known for the one-shot method. On the other hand, the altitude estimation method described in Section 2.2 regards the point closest to the two lines of sight as the target point. The geolocation error caused by attitude measurement errors is more concentrated on the target altitude because the horizontal distance between the UAV and the target is much smaller than the vertical distance. Finally, the distance error of our method with eight images is smaller than that of the one-shot method, because the distance error contains the three-dimensional errors.

Then, we assumed that only the position has Gaussian measurement error in the UAV’s navigation state, and then compared the performance of the proposed framework and the one-shot method. Specifically, we set ${\sigma }_z=1m,\dots ,5m$, respectively, and ${\sigma }_x={\sigma }_y={\sigma }_z$. In the comparisons, we compare the three-dimensional geolocation errors and the distance error, as shown in Figure 10. The same conclusion can be obtained; that is, our method can greatly mitigate the impact of Gaussian measurement errors on the geolocation results due to the using of the historical measurement data. The difference is that our method with eight images is better than the one-shot method in terms of z coordinate. There are two reasons for this phenomenon. On the one hand, the influence of UAV’s position measurement error is greater than that of attitude measurement error for the one-shot method. On the other hand, the proposed framework can mitigate the negative influence of Gaussian measurement error by using weighted least-squares. In terms of distance error, our method is superior to the one-shot method, no matter whether four images or eight images are used.

After that, we assume that both the position and the attitude angle have Gaussian measurement error in the UAV’s navigation state, and then compare the performance of the proposed framework and the one-shot method. Specifically, we set ${\sigma }_z=1m,\dots ,5m$, ${\sigma }_{\psi }=1^{\circ },\dots ,5^{\circ }$, respectively, and ${\sigma }_x={\sigma }_y={\sigma }_z$, ${\sigma }_{\psi }={\sigma }_{\theta }={\sigma }_{\phi }$. It can be seen in Figure 11 that our method with eight images has better mitigation effects on the negative influence of various Gaussian measurement errors. In this experiment, the assumption of measurement errors is consistent with most application scenarios. Even if the yaw-angle bias is not considered, the proposed framework is better than the one-shot method in practical applications.

In addition to the Gaussian measurement errors, the yaw-angle biases provided by the low-quality sensors also affect the geolocation results. Therefore, we assume that the navigation state has both Gaussian measurement errors and yaw-angle biases, and compare the performances of the proposed method and the one-shot method at different yaw-angle biases. In detail, we set ${\sigma }_{\psi }={\sigma }_{\theta }={\sigma }_{\phi }=2^{\circ }$ and ${\sigma }_x={\sigma }_y={\sigma }_z=3m$. The yaw-angle biases $\delta {\psi }_L,\delta {\psi }_R$ of the left and right UAVs are set as (34). We set $\delta {\psi }_0$ to 10, 15, 20, 25, and 30, respectively, and then compared the performances of the two methods. It is shown in Figure 12 that the geolocation accuracy of the one-shot method is greatly reduced when there is bias in yaw-angle measurement, and as the yaw-angle measurement bias increases, the geolocation accuracy of the one-shot method becomes lower and lower. Compared with the one-shot method, the accuracy of our method does not change significantly as the yaw-angle measurement bias increases. This is because our method can estimate the yaw-angle bias through the iterations between the estimation of the target’s altitude and the parameter regression to avoid the negative impact of the yaw-angle measurement bias on geolocation accuracy. However, in terms of z-coordinate error, the accuracy of both methods has no obvious change as yaw-angle bias increases. There are two reasons for this phenomenon. On the one hand, the one-shot method assumes that the distance between the UAV and the moving target is known and the yaw-angle measurement error has little impact on the target’s altitude estimation. On the other hand, the proposed framework can eliminate the negative influence of the yaw-angle biases by utilizing the iterations between the processes of altitude estimation and parameter regression.

In conclusion, we first verified the effectiveness of the proposed corresponding-point-matching method, target altitude estimation method, and yaw-angle bias estimation method in the simulation environment. In addition, we conducted a lot of simulation experiments to verify the effectiveness and robustness of the proposed framework, and the simulation experiments showed that the geolocation results of the proposed framework are more accurate and stable under Gaussian measurement errors and yaw-angle biases compared with the commonly used one-shot method.

3.2. Evaluation in Real Environment

A real indoor experiment has also been performed to further validate the proposed framework. The UAVs used in this experiment were a laboratory product designed by our group, as shown in Figure 13a. A camera waws installed vertically downward on each UAV, so the attitude of the camera could be known from the attitude of the UAV. The two UAVs tracked the moving target, as shown in Figure 13b, and transmitted their pose information and target images to the ground station in real time. The two UAVs and the ground station were run in the ROS system, and the precise position and attitude information of the UAVs were provided by VICON (a motion capture system).

The important experimental parameters are shown in

Table 5. Experimental parameters.

Number of Images	Flight Altitude	Flight Speed	${\mathit{\sigma }}_{\mathit{\psi }}$	${\mathit{\sigma }}_{\mathit{z}}$	${{\mathit{\delta }}_{\mathit{\psi }}}_{\mathit{L}}$	${{\mathit{\delta }}_{\mathit{\psi }}}_{\mathit{R}}$
8	2.8 m	0.26 m/s	$2^{\circ }$	0.2 m	${10}^{\circ }$	$-{10}^{\circ }$

. The results of the realistic experiment are shown in

Table 6. Geolocation results.

Coordinate	MAE $\left(\mathit{m}\right)$	STD $\left(\mathit{m}\right)$	MAX $\left(\mathit{m}\right)$
X	0.035	0.029	0.125
Y	0.034	0.025	0.104
Z	0.159	0.113	0.446

. A total of 147 geolocation estimates were performed while the UAVs were tracking the moving target. Each geolocation needed to use eight images, among which, the distance between two adjacent images should satisfy $T_{re}\ge 0.25$ m. This shows that our method successfully achieved the geolocation of the moving target, and the absolute mean errors of the three coordinates were 0.035, 0.034, and 0.159 m, respectively. When taking advantage of eight images for geolocation, the FPS was 22, which still meets the real-time requirements.

The geolocation path of the realistic is shown in Figure 14. The black path represents the actual position of the moving target obtained from VICON. The yellow path represents the position of the moving target obtained from our method. It can be seen that the path obtained by our method is undulating due to the influence of Gaussian measurement error of the UAV’s navigation state. The above-mentioned simulation experiments have demonstrated that our method can mitigate the effects of Gaussian measurement errors by utilizing the historical measurements.

The simulated and realistic experiments show that the proposed framework implements the function of geolocating the moving target using only UAV vision and does not rely on the accurate AHRS. Compared with the commonly used one-shot method, the proposed framework can mitigate the effects of measurement errors in a UAV’s position and attitude by using multiple measurement data and estimating the yaw-angle biases.

4. Discussion

In order to solve the problem that Gaussian measurement errors and system measurement errors of UAV state will lead to a decline in moving-target geolocation accuracy, we proposed a moving-target geolocation framework based on two UAVs. Our method uses a two-step strategy to obtain the final result. The accuracy of the final result depends on three aspects: the matching accuracy for the corresponding points, the accuracy of estimating target altitude, and the accuracy of estimating yaw-angle biases. It can be seen in the experimental results shown in Figure 8 and Table 1 that the proposed corresponding-point-matching method and error-corresponding-point-filtering method can enable historical measurement data to be used to estimate the current position of a moving target. In the subsequent processing step, our method obtains the final result through multiple iterations between target altitude estimation and yaw-angle bias estimation. The experimental results shown in Table 2 and Table 3 demonstrate that our method can estimate the target altitude and yaw-angle biases, and the more measurement data are used, the more accurate the estimated target altitude and yaw-angle biases will be. Even so, due to the influence of Gaussian measurement errors, it is still difficult for the proposed framework to accurately estimate the yaw-angle biases, which will be one of the directions of our future exploration.

After that, we evaluated the geolocation performance of the proposed method. For Gaussian measurement errors, obtaining more measurement data is the most effective method to mitigate the influence of Gaussian measurement errors. However, we do not simply increase the number of UAVs to obtain more observation data. If so, the cost will be higher and the system will be extremely complex. Our strategy is to have historical measurement data used to estimate the current position of the moving target through corresponding point matching. From the experimental results in Figure 9, Figure 10 and Figure 11, we can see that our method can mitigate the influence of Gaussian measurement errors very well. In contrast, the one-shot method is easily affected by the Gaussian measurement error because it can only use one measurement datum. For yaw-angle biases, our strategy is to eliminate the negative impact of yaw-angle biases through multiple iterations between target altitude estimation and yaw-angle bias estimation. The experimental results in Figure 12 show that our method can greatly mitigate the impact of yaw-angle biases on geolocation results compared with the one-shot method, which does not consider yaw-angle biases. It is worth noting that Zhang et al. [24] also considered the influence of yaw-angle biases on target geolocation, but their method assumes that the target is stationary. Unlike this, our method can be applied to moving-target geolocation.

In practical applications, the state measurement error of UAV is an important factor affecting the geolocation accuracy of moving-target tracking. The experimental results show that the proposed method can effectively mitigate the influence of UAV state measurement error on moving-target geolocation accuracy. However, the limitation of our method is that corresponding point matching requires the use of image background information, which is difficult to achieve in some scenes with simple background information (such as the geolocation of a moving target on the sea). Therefore, our next research plan is to develop a more general geolocation method.

5. Conclusions

In this paper, we explored the problem of moving-target geolocation based on a UAV platform with low-quality navigation state sensors. Our aim was to mitigate the influences of Gauss measurement errors and yaw-angle measurement bias from low-quality sensors on the moving-target geolocation accuracy. For this challenge, we proposed a vision-based moving-target geolocation framework using dual UAVs, which can use historical measurement data to estimate the yaw-angle biases of two UAVs and mitigate the impact of Gaussian measurement errors. The results of the simulations and realistic experiments demonstrate that the proposed framework has higher geolocation accuracy than the traditional one-shot based method.

The framework proposed in this paper is only applicable to single target geolocation scenarios. However, multi-target geolocation based on the UAV platform is also one of the urgent needs in practical applications. Therefore, in future work, we will develop a more general geolocation method.