Quadcopter Drone for Vision-Based Autonomous Target Following

Abstract

Unmanned aerial vehicles (UAVs) are becoming popular in various applications. However, there are still challenging issues to be tackled, such as effective obstacle avoidance, target identification within a crowd, and specific target tracking. This paper focuses on dynamic target following and obstacle avoidance to realize a prototype of a quadcopter drone to serve as an autonomous object follower. An adaptive target identification system is proposed to recognize the specific target in the complicated background. For obstacle avoidance during flight, we introduce an idea of space detection and use it to develop a so-called contour and spiral convolution space detection (CASCSD) algorithm to evade obstacles. Thanks to the low architecture complexity, it is appropriate for implementation on onboard flight control systems. The target prediction is integrated with fuzzified flight control to fulfill an autonomous target tracker. When this series of technical research and development is completed, this system can be used for applications such as personal security guard and criminal detection systems.

1. Introduction

Over the past few years, various innovative applications of unmanned aerial vehicles (UAVs) have emerged due to lower cost in hardware and appearance of powerful onboard microcomputers.

UAVs can conquer topographic barrier so that they are appropriate for conducting missions in dangerous zones, replacing manpower to transport goods, or even serving as a personal standby assistant.

Recently, artificial intelligence (AI)-based automatic control technology has received attention attributed to the significant improvement such as the generalization capability of deep learning. Deep learning drives many AI applications that improve automation, performing analytical and physical tasks without human intervention. For instance: the robot “pepper” in [1] is used for domestic and navigation purposes; the biped robot is used to replace soldiers [2]; there are medical hospitality robots [3], autonomous cars [4,5], and much more. In addition to land vehicles, there are various aerospace vehicles developed for specific purposes, such as the delivery UAVs [6,7,8], disaster relief UAVs, and multi-machine protocol flying UAVs [9].

UAVs that track targets need to adapt to various environments to fly. In [10], the authors proposed a robust adaptive recursive sliding mode attitude control, which can make the quadrotor effectively resist unknown interference. The work of [11] proposed an anti-jamming control scheme based on multi-observers to counteract multiple interferences so that the quadrotor can resist strong wind and load interference.

Focusing on the technical issues, while research efforts in target following and obstacle avoidance techniques have been quite mature in the field of robotics, the two issues are still challenging for the operational scenarios of miniature UAVs such as quadcopter drones because of the relatively limited hardware resources available and the resulting neutral instability [12].

Improving the reliability of UAVs is one of the challenging issues in the market. Crash events of UAVs are not unusual. Moreover, static or dynamic target identification in a crowd is also a difficult issue for miniature UAV development due to the limited computational resources and budget cameras used [13]. In the academic research field, while there are research efforts dedicated to investigating target tracking [14,15,16] and obstacle avoidance [17,18,19,20], further improvement on functional enhancement, especially in target identification and tracking, is still highly demanded. For example, the proposed target tracking in [15] requires the target to be identified wearing a special T-shirt with a QR code on it. The study of [16] relies on color features to identify the specific target. The rough obstacle avoidance method adopted in [19] might cause the UAV to collide with obstacles when the sensor is not in parallel with the obstacles. Although the research tasks of [21,22] have combined the capability of target tracking and obstacle avoidance, their results were only verified via simulation study. The common difficulty of various research efforts is that the extremely large computational resources were unsuitable for implementation on the onboard microprocessor.

Considering the general weaknesses of the current approaches, this research task proposes the methods developed for target identification and movement prediction, the adaptive cruise control (ACC) system, and the Kalman filtering navigation to fulfill a mission-oriented flight control system. Our research motivations are listed as follows:

• The tracking capability of the commercial miniature UAV is still immature [12,13]. The methods developed in the papers might not be applicable in real-world applications [14,15]. These methods are feasible, but are slightly restricted from the viewpoint of practical applications. For human target following, this research proposes an adaptive target identification system to resolve the problem when the specific target is in a crowd.
• The current UAV obstacle avoidance algorithms revealed in the papers were difficult to implement in the small-scale embedded system because of the large computational sizes. Here, we propose a novel contour and spiral convolution space detection (CASCSD) algorithm to tackle the issue. Through the emulated expansion and etching of the image processing, we can filter out noises while enlarging imaging signals to indicate that if the obstacles might interfere with the flight path. This algorithm consumes less computational resources and is appropriate to be used in the current miniature UAV applications.
• Intelligent mobile assistants have recently become popular; however, UAVs moving inside a building, on stairs, or in rugged areas is still a challenge. A miniature UAV drone is one of the potential substitutes for work under these scenarios.

In addition, in order to improve the research quality, we also refer to other UAV and tracking-related literature, such as [23], which mainly discusses the deep learning application of UAVs in Internet of Things applications and investigates the related applications of UAVs in detail. Ref. [24] discusses the issues related to the bionic control of UAVs, which can provide more ideas for the control of UAVs in this paper. In the research of tracking, [25] uses a UAV to track the landing point on the roof of a car and allows the UAV to safely land at the target point. Ref. [26] proposed an adaptive lightweight UAV tracking algorithm, which allows UAVs to track various targets at high altitudes. Ref. [27] proposed a distance estimation method based on hybrid stereo vision, which can accurately calculate the distance between the robot and the object and can provide more ideas for this research on obstacle avoidance. Ref. [28] used deep learning, simulated tracking, and obstacle avoidance with multiple UAVs, and the results showed the algorithm’s effectiveness. Synthesizing various research ideas, this study found that the ability of UAVs to fly indoors and outdoors is still lacking, and this research intends to complete a personal security guard system. The overall system of UAVs needs to have identification, tracking, and obstacle avoidance at the same time. The three functions and the algorithm need to be as lightweight as possible. Only by fulfilling these demands can the experimental purpose of the research be achieved.

2. System Description

The dynamics of the UAV directly affect flight stability [29]; to enable the UAV to have intelligent tracking and obstacle avoidance functions, we first need to establish a quadrotor flight dynamic model and control equations. Secondly, the image and the entire coordinate system must be convertible so that we can import the image information and various algorithms into the flight control perform tasks.

2.1. Architecture

Figure 1 illustrates the quadcopter architecture implemented in this research task. The miniature UAV drone is designed to be a vision-based autonomous target follower. The functions of the key modules include the following modules: a flight control system including an inertial navigation system (INS), gyroscope, optical flow sensor using ground texture and visible features for indoor positioning, and barometer for attitude determination; an onboard AI processor is responsible for intelligent target identification and tracking, flight path computation with obstacle avoidance, and realization of all supporting algorithms; and a depth camera is responsible for transmitting external images to the AI processor. It also provides information of the relative distance between the target and the environment.

2.2. Modeling

The four-rotor mathematical model of the miniature quadcopter for the purpose of target identification and tracking is given in Appendix A.

Flight Dynamics Model (FDM) Description

The world frame and the quadcopter body frame of this study are shown in Figure 2, where $(X_E,Y_E,Z_E)$ is the inertial coordinate system, $O_E$ is the origin of the inertial coordinate, and $(e_1,e_2,e_3)$ is the unit vector along the inertial coordinate axis. $(X_{UB},Y_{UB},Z_{UB})$ is the rigid body coordinate system of the quadcopter body, $O_{UB}$ is the origin of the rigid body coordinate, and $(b_1,b_2,b_3)$ is the unit vector along the rigid body coordinate axes.

Proceeding to the development given in the Appendix A, we have the acceleration and angular acceleration equations of the quadcopter as:

$$\begin{array}{l}\ddot{x}=\frac{1}{M}{k}_2{G}_{R^3}(\overrightarrow{b_3},\overrightarrow{e_1})({\omega }_1^2+{\omega }_2^2+{\omega }_3^2+{\omega }_4^2),\\ \ddot{y}=\frac{1}{M}{k}_2{G}_{R^3}(\overrightarrow{b_3},\overrightarrow{e_2})({\omega }_1^2+{\omega }_2^2+{\omega }_3^2+{\omega }_4^2),\\ \ddot{z}=\frac{1}{M}(k_2{G}_{R^3}(\overrightarrow{b_3},\overrightarrow{e_3})({\omega }_1^2+{\omega }_2^2+{\omega }_3^2+{\omega }_4^2)-Mg),\\ \dot{p}=\frac{1}{I_{11}}\ell k_2({\omega }_1^2+{\omega }_4^2-{\omega }_2^2-{\omega }_3^2)\\ \dot{q}=\begin{array}{l}\frac{1}{I_{22}}\ell \hfill \end{array}{k}_2({\omega }_3^2+{\omega }_4^2-{\omega }_1^2-{\omega }_2^2),\\ \dot{r}=\begin{array}{l}\frac{1}{I_{33}}\ell \hfill \end{array}{k}_1({\omega }_1^2+{\omega }_3^2-{\omega }_2^2-{\omega }_4^2)\end{array}$$

(1)

To proceed, we perform coordinate transformation from the body frame to the world frame along the yaw, roll, and pitch axes:

$$\left[\begin{array}{ccc}c\psi & -s\psi & 0\\ s\psi & c\psi & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}c\theta & 0& s\theta \\ 0& 1& 0\\ -s\theta & 0& c\theta \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& c\varphi & -s\varphi \\ 0& s\varphi & c\varphi \end{array}\right]=\left[\begin{array}{ccc}c\psi c\theta & c\psi s\theta s\varphi -s\psi c\varphi & c\psi s\theta c\varphi +s\psi s\varphi \\ s\psi c\theta & s\psi s\theta s\varphi +c\psi c\varphi & s\psi s\theta c\varphi -c\psi s\varphi \\ -s\theta & c\theta s\varphi & c\theta c\varphi \end{array}\right]$$

(2)

where $\theta ,\varphi ,\psi$ denote the angles of pitch, roll, and yaw respectively (see Figure 3). For completeness, the coefficients of the air resistance $C_{D1},C_{D2},C_{D3}$ and coefficients of the rotary resistance $C_{1DR},C_{2DR},C_{3DR}$ are considered. It can then be further transferred into the state space representation by considering the gyro effect:

$$\begin{array}{l}\ddot{x}=\frac{1}{M}(k_2(c\psi s\theta c\varphi +s\psi s\varphi )u_1-C_{D1}\dot{x}),\\ \ddot{y}=\frac{1}{M}(k_2(s\psi s\theta c\varphi -c\psi s\varphi )u_1-C_{D2}\dot{y}),\\ \ddot{z}=\frac{1}{M}(k_2(c\theta c\varphi )u_1-Mg-C_{D3}\dot{z}),\\ \dot{p}=\frac{1}{I_{11}}(\ell k_2{u}_2-C_{1DR}p)-\frac{I_{33}-I_{22}}{I_{11}},\\ \dot{q}=\begin{array}{l}\frac{1}{I_{22}}\hfill \end{array}(\ell k_2{u}_3-C_{2DR}q)-\frac{I_{11}-I_{33}}{I_{22}},\\ \dot{r}=\begin{array}{l}\frac{1}{I_{33}}\hfill \end{array}(\ell k_1{u}_4-C_{3DR}r)-\frac{I_{22}-I_{11}}{I_{33}}\end{array}$$

(3)

3. Target Tracking and Obstacle Avoidance

A depth camera is utilized to detect the target and surrounding objects. Before conducting target identification, image calibration is conducted.

3.1. Calibration and Coordinate Conversion

The pinhole imaging model [30,31] is used to describe this process, and it is corrected by the principle of similar triangles (see Figure 4).

Suppose that the camera coordinate system is denoted by $(X_H,Y_H,Z_H)$ with the origin $O_H$. The pixel coordinates are ${X^{\prime }}_H,{Y^{\prime }}_H,{Z^{\prime }}_H$ and the origin is ${O^{\prime }}_H$, as shown in Figure 4. Furthermore, there is a targeted point $P(x_h,y_h,z_h)$ in reality, and its new coordinate mapped to the pixel plane is $P({x^{\prime }}_h,{y^{\prime }}_h,{z^{\prime }}_h)$. Let the focal length f be the distance from the pinhole to the pixel coordinate plane. This gives rise to the following ratio formula:

$${X^{\prime }}_H=f\frac{X_H}{Z_H},\hspace{1em}{Y^{\prime }}_H=f\frac{Y_H}{{\mathrm{Z}}_H}$$

(4)

We define the origin of the pixel coordinate system $c$ as being located at the upper left corner of the image; the $U_H$ axis is parallel to the $X_H$ axis, and the $V_H$ axis is parallel to the $Y_H$ axis and is at the same plane as the pixel coordinate system. Thus, there is a two-axis difference between the two-pixel coordinates.

Assume that the $U_H$ and $V_H$ axes are zoomed in by $\alpha$ and $\beta$ times, respectively, and the origin is linearly shifted by the base unit ${\left[c_x,c_y\right]}^T$. One obtains:

$$U_H=\alpha {X^{\prime }}_H+c_x,\hspace{1em}{V}_H=\beta {Y^{\prime }}_H+c_y$$

(5)

Let $\alpha f=f_x$ and $\beta f=f_y$, then

$$U_H=f_x\frac{X_H}{Z_H}+c_x,\hspace{1em}{V}_H=f_y\frac{Y_H}{Z_H}+c_y$$

(6)

Transforming this into a matrix representation gives

$$Z_H\left[\begin{array}{c}{U}_H\\ V_H\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_H\\ Y_H\\ Z_H\end{array}\right]=KP$$

(7)

where $K$ is the camera’s inner parameter matrix. It is used to compute the camera’s outer parameter matrix.

Let $P_w$ be the point P from the camera coordinate to the world coordinate, where R and t are the rotation and translation of the camera, respectively.

$$Z_H\left[\begin{array}{c}{U}_H\\ V_H\\ 1\end{array}\right]=K(R{P}_w+t)$$

(8)

Performing normalization gives

$$R{P}_w+t\to {\left[\begin{array}{ccc}\frac{X_H}{Z_H}& \frac{Y_H}{Z_H}& 1\end{array}\right]}^T$$

(9)

The point P is at the normalized plane at the point $Z_H=1$. The normalized coordinate that multiplies the camera intrinsic parameters results in the modified pixel coordinate.

3.2. Dynamic Target ID and Locking

The traditional YOLO model [32] can only identify objects that are already trained, it would be insufficient while being applied to the current objective. We updated the function of specific target identification by considering the limited onboard resources.

At first, the results $b_w$, $b_h$, $b_x$, $b_y$ of the bounding boxes generated by the modified YOLO network given in [33] are used to determine the centroid position of the targeted person. The coordinates of the four vertices of each bounding boxes are $(b_x-\frac{b_w}{2},b_y-\frac{b_h}{2}),$ $(b_x+\frac{b_w}{2},b_y-\frac{b_h}{2}),$ $(b_x-\frac{b_w}{2},b_y+\frac{b_h}{2})$, and $(b_x+\frac{b_w}{2},b_y+\frac{b_h}{2})$. To be realized on the onboard computer, the modified YOLO is only used to frame possible objects in the image frame.

To track the specific object with the limited computational resources, a lightweight algorithm is proposed in this research task for dynamic target identification. The algorithm involves three parts which are calculated per video frame, including: the color scores of the target’s clothes, pants, and hair; the respective length-to-width ratio scores; and the tracking error of the dynamic target. The equations are described as follows:

$$\begin{array}{l}CS={\displaystyle \sum _{i=1}^n{w}_{1i}\cdot component color\left[i\right]},\\ LS={\displaystyle \sum _{i=1}^m{w}_{2i}\cdot component length\left[i\right]}\end{array}$$

(10)

where component color $\left[i\right]$ denotes the ith colored component with a Boolean value (“1” for matched and “0” for unmatched), component length $\left[i\right]$ denotes the ith length-to-width ratio component (“1” for matched and “0” for unmatched), $m$ and $n$ represent the total number of colored components and length-to-width ratio components, respectively, and $w_{1i}$ and $w_{2i}$ are the ith weighted exponents. After obtaining scores of CS and LS, the position score is calculated according to the target locking schematic illustrated in Figure 5 with the score

$$PS=A\frac{1-e^{-s(PA-bias)}}{1+e^{-s(PA-bias)}}$$

(11)

where $A=CS+LS$, PA refers to the accuracy of calculating the target position of each frame. $bias$ and $s$ are the offset and slope of the sigmoid function, respectively, with their respective equations being as follows: $PA={\left(\sqrt{{\left(x_p-x_c\right)}^2+{\left(y_p-y_c\right)}^2+{\left(z_p-z_c\right)}^2}\right)}^{-1}$, where $\left(x_p,y_p,z_p\right)$ denotes the position of the current target frame, $\left(x_c,y_c,z_c\right)$ denotes the position of the previous frame, and $bias=\raisebox{1ex}{$n_1$}\!\left/ \!\raisebox{-1ex}{$IS$}\right.$, where $n_1$ represents the number of targets locked within the image space $IS=L\cdot W$, with L and W being the image length and width, respectively. The bias is used to adjust the maximum error size of the position and filter out the surrounding noise. For the value of $s$, when the target position falls within the gray area as shown in Figure 5, it means that the target position is not located in the central focused region. The closer the position is to the edge, the lower the score. It is designed by $s=\raisebox{1ex}{$RS$}\!\left/ \!\raisebox{-1ex}{$m_1$}\right.$, where $m_1$ represents the number of people locked within the reconnaissance space $RS(=\pi ab)\subset IS$ with the semi-major axis of length $a=\alpha W$ and semi-minor axis of length $b=\beta L$ and $0<\beta \le \alpha \le 1$. The rules are summarized as follows:

• There are many people that appear in the image. The bias is increased to reflect the noisy background.
• There are few people in the image, but they are tightly crowded. The bias is decreased and s is increased by increasing $\beta ,\alpha$ to rise discriminative sensitivity.
• There are many people, but they are widely dispersed. The bias is decreased to aid highlighting the target.
• There are few people and they are widely dispersed. The bias is set to zero and a large s is suggested to boost the discriminative effect.

After obtaining $CS$, $LS$, and $PS$, the target ID score is characterized by

$$RR=\left(C{S}^{\prime }{W}_1+L{S}^{\prime }{W}_2+P{S}^{\prime }{W}_3\right)\cdot 100\%$$

(12)

where the normalized individual scores $C{S}^{\prime }$, $L{S}^{\prime }$, and $P{S}^{\prime }$ correspond to $CS$, $LS$, and $PS$, respectively, and $W_i,\hspace{0.17em}i=1~3$ are the respective weighting factors satisfying ${\displaystyle \sum _{i=1}^3{W}_i}=1$.

Figure 6 illustrates the operational flow of the adaptive target identification system.

3.3. Target Positioning in 3D Space

Once the target position in the pixel coordinates is obtained, it is transferred to the Cartesian plane. In addition, the relative distance D with respect to the target provided by the depth lens is integrated with the pixel coordinate representation to yield the target position in the spherical coordinates, as shown in Figure 7.

For the current experiment, the camera’s field of view (FOV) is 69.4 × 42.5 × 77 degrees with 640 × 480 p (4:3) pixel resolutions. It is used to compute the degree of the pixels within a frame. We end up with the following equations to transform the spherical coordinate system to the Cartesian coordinates:

$$\begin{array}{l}{X}_T=D\cos ({\theta }_p)\sin ({\theta }_y),\\ Y_T=D\cos ({\theta }_p)\cos ({\theta }_y),\\ Z_T=D\cos ({\theta }_y)\sin ({\theta }_p)\end{array}$$

(13)

where ${\theta }_p$ and ${\theta }_y$ are the pitch and yawing plane line-of-sight (LOS) angles, respectively. The quadcopter rotates its body to move; thus, we have

$$P_{new}=R{P}_{origin}$$

(14)

where $P_{new}$ is the quadcopter position after performing rotation and R is the coordinate rotational matrix.

3.4. Target Movement Estimation

Let the target state vector be $x_t$, including the target position p, the velocity v, and the acceleration a, which is random and conformed to the Gaussian distribution with mean $\mu$ and variance $\Sigma$. Consider external interference and measurement so that the prediction of measurement statistics is characterized by the estimated measurement ${\widehat{z}}_t=H_t{\widehat{x}}_t$, with the error covariance of estimation $P_t$ and t denoting the current time step and the real measurement characterized by $({\mu }_1,{\sum }_1)=(z_t,R_t)$, where $R_t$ is the measurement noise covariance.

The prediction equations of the state and error covariance $P_t$ are given by

$$\begin{array}{l}{\widehat{x}}_t=F_t{{\widehat{x}}^{\prime }}_{t-1},\\ P_t=F_t{P}_{t-1}{F}_t{}^T+Q_t\end{array}$$

(15)

where the process noise covariance $Q_t={\epsilon }_q{I}_9$ and the system matrix $F_t$ given by

$$F_t=I_9+\left[\begin{array}{ccc}{0}_3& \delta t{I}_3& 0.5\delta t^2\\ 0_3& 0_3& \delta t{I}_3\\ 0_3& 0_3& 0_3\end{array}\right]$$

where $\delta t$ is the sampling period. Including the two Gaussian distribution terms provides the state update equations:

$$\begin{array}{l}{{\widehat{x}}^{\prime }}_t={\widehat{x}}_t+K^{\prime }(z_t-{\widehat{z}}_t),\\ P_t{}^{\prime }=P_t(I_9-K^{\prime }{H}_t),\\ K^{\prime }=P_t{H}_t{}^T(H_t{P}_t{H}_t{}^T+R_t)-1\end{array}$$

(16)

where $H_t=\left[\begin{array}{cc}{I}_3& 0_{3\times 6}\end{array}\right]$ and $R_t={\epsilon }_r{I}_3$, with ${\epsilon }_r$ being a small positive constant.

3.5. Flight Path Planning

The imaging processing system uses an embedded microcomputer to realize deep learning and Kalman filtering computation. We first propose a contour and spiral convolution space detection algorithm (CASCSD) to handle obstacle avoidance and plan an appropriate flight path to track the target with its center of mass located at $O_S(x_{s,}{y}_s,z_s)$.

Using the flyable space detection process, the image is divided into two types of blocks, denoted by “1” or “0”, which refer to the safe-to-fly (STF) and non-STF regions, respectively. We use the infrared of the depth camera to measure the relative distance. As it is easily affected by environmental light, the image processing system first slightly erodes the scope of the space, that is, the expanded scope of the obstacle, which is performed to prevent smaller obstacles from being filtered. Next, it enhances the dilated space range to filter out the light pollution or other noise that infrared ray cannot measure. Then, the space is eroded back to its original size; see Figure 8 for the process. A safety buffering range to prevent the UAV from drifting is reserved at the edge of each detected obstacle.

To ensure that the target can be tracked with the shortest path, the two-dimensional pixel coordinates are inversely derived from the predicted coordinates generated by the Kalman filter. Let the predicted coordinates of the Kalman filter be $O_P={\left[x_p,y_p,z_p\right]}^T$. The line-of-sight (LOS) angles ${\theta }_p$ and ${\theta }_y$ can be deduced from

$${\theta }_p={\cos }^{-1}(\frac{z_p}{D_p}),\hspace{1em}{\theta }_y={\cos }^{-1}(\frac{x_p}{D_p})$$

(17)

where $D_P=‖O_p‖$. Based on the current camera’s FOV, the center position of the target measured from the image are $c_x=11.3{\theta }_p\hspace{0.33em}\mathrm{and}\hspace{0.17em}{c}_y=9.22{\theta }_y$.

The spiral convolution from the centroid position outwards determines whether there is available space for the quadcopter to fly forward. The spiral convolution means that the filter matrix rotates from inside to outside and convolves the spatial detection feature map. The size of the filter depends on the coverage of the boundary on the imaging plane, as illustrated in Figure 9.

As shown in Figure 9, the image projection will be zoomed in. Therefore, it is necessary to set up a threshold of the mapping frame from the FOV. For the current system, the lengths of $x$ and $y$ axes per pixel are determined by $imag{e}_{width}=\raisebox{1ex}{$threshold\tan (34.7)$}\!\left/ \!\raisebox{-1ex}{$240$}\right.$ and $imag{e}_{height}=\raisebox{1ex}{$threshold\tan (21.25)$}\!\left/ \!\raisebox{-1ex}{$320$}\right.$.

The control system measures the distance between the camera and the global positioning system (GPS), the left and right wings, and the feet and the ratio between them, and calculates the size of the quadcopter on the imaging plane at the critical distance.

For the spiral convolution, it starts from the target mass center on the pixel coordinates to find the space to move from the inside to the outside. If the target is not found, the next process is completed. Because the space around the target has already been searched during the process of spiral convolution, there is no need to continue searching for the shortest path around the target. Instead, it directly starts searching for the remaining part.

4. Adaptive Cruise Control (ACC)

The adaptive cruise control system aims at the distance between the quadcopter drone and the target by adaptive adjusting the speed of target tracking.

4.1. Thrust Force

The acceleration of the quadcopter is determined by the body inclination angle $a=g\tan (\theta )$. As the moment of inertia $I_{xx}$, arm length $\ell$, and motor coefficient $k_2$ are all constants, thus the rotational speed of the motor is a major concern. The thrust force is computed by $‖F‖=23{(\frac{length}{10})}^3\frac{pitch}{10}{(\frac{volt\cdot kV}{1000})}^2$, where length is the propeller diameter, pitch denotes the propeller pitch angle, and $kV$ denote the voltage per revolution of the motor. The thrust force of the motor is $1392$ gram-force. With the obtained thrust force, propeller diameter, and moment of inertia, the inclination is $\theta (t)={\theta }_0+0.5\ddot{\theta }{t}^2$.

4.2. Fuzzy Control Implementation

Figure 10 illustrates the scenario of the target follower. When the quadcopter is far away from the target, the flight control command increases the angle of the pitch axis to speed up the task.

As the quadcopter drone gradually approaches the target with higher speed, the relative distance between the target and quadcopter would be gradually decreasing. When the quadcopter decelerates but the target person starts to move forward, the quadcopter would respond by speeding up again. The faster the target walks, the faster the quadcopter will follow.

Figure 11 displays the schematic diagram of the overall control system where the motor mixing algorithm (MMA) transfers the roll, yaw, and pitch control commands to the driving commands of four DC brushless motors.

To cope with the high nonlinearity of the quadcopter flight dynamics, a fuzzy logic control is incorporated here as the core of the ACC system. A fuzzy guidance controller with two inputs and one output is implemented. The adopted membership functions are shown in Figure 12, Figure 13 and Figure 14, where the fuzzified inputs are $\mu \left(D\right)$ and $\mu \left(V\right)$ and the fuzzified output is $\mu \left(O\right)$, with $\mu \left(.\right)$ representing the degree of membership of fuzzification; $D$ and V are the relative distance and speed between the quadcopter and the target, respectively, and $O$ is the acceleration output. Each group of the membership functions possesses seven fuzzy sets. The membership functions in Figure 12 specify the relative distance between the quadcopter and the target. Considering the capability of the camera, the fixed relative distance is set to be 3.5 m. This corresponds to the linguistic variable $ZO$.

The membership functions in Figure 13 are used to fuzzify the relative speed. In the universe of discourse, zero means that the target and the quadcopter are in still status or they are synchronously moving; the interval from zero to two refers to the walking status of the target, and the interval from two to four refers to the running status. Refined fuzzification of the variables is achievable by increasing the number of membership functions.

Figure 14 shows the membership functions characterizing the acceleration command. The goal is for the quadcopter to maintain a certain distance and quadcopter speed and LOS between to the target.

There are 25 guidance rules in total that are sorted in

Table 1. Fuzzy guidance rule base.

O				D
		Nb	Ns	Zo	Ps	Pb
	nb	Pb	Pb	Ps	Ps	Zo
	ns	Pb	Ps	Ps	Zo	Ns
V	zo	Ps	Ps	Zo	Ns	Ns
	ps	Ps	Zo	Ns	Ns	Nb
	pb	Zo	Ns	Ns	Nb	Nb

. The guidance law adopts proportional navigation guidance (PNG) commonly adopted by homing air target missiles based on the fact that the target and quadcopter are on a contact course when their direct LOS does not significantly change direction as the range closes. The guidance rules here are to keep a constant relative distance between the target and the quadcopter. The center of gravity method is adopted for the defuzzification of the inferred results.

5. Experimental Verification

In order to prove the feasibility of the theory, we conducted experiments such as outdoor tracking targets, indoor tracking targets, ladder tracking targets, and intelligent obstacle avoidance. The results are shown and explained as follows, and one can also refer to the following video URL: https://www.youtube.com/watch?v=ko9BaSFTcwA (accessed on 21 November 2021).

5.1. Autonomous Flight

Firstly, to examine the capability of a fundamentally stable flight, a 3D rectangular flight for the performance validation of the ACC was conducted. The real-world experimental results are shown in Figure 15 for the autonomous flight. The green solid line shows the planned flight path, the blue dotted line shows the flight result estimated by the extended Kalman filter (EKF), and the red dotted line presents the GPS measurement result. It verifies that under the interference of strong wind, the offset of the UAV flight path and the planned path is satisfactorily less than 0.2 m under the help of the real-time kinematic positioning (RTK) technique [23].

5.2. Target Identification

A demonstrative image for locating the object based on the proposed YOLO algorithm is shown in Figure 16, where the matching scores are shown corresponding to the target feature, human face, clothing, and shorts.

The setting of weighting factors for target features is $W_1:W_2:W_3=0.3:0.2:0.5$. IS is set to 4 × 3, a = 0.8, and b = 0.6. The position accuracy is normalized via the sigmoidal function, and $(x_p,y_p,z_p)$ are the prediction coordinates from the previous frame of the target. It is based on the prediction of the position from the last frame compared with the measured position in the current frame. If the target achieves a high score with an accuracy level higher than, for example, 70%, the subject is recognized. We aimed to track the target with an orange jacket, blue trousers, and short black hair, as illustrated in Figure 17. The experimental result of the recognition rate (RR) of each subject within the video frame is listed in

Table 2. RR of the objects that appear in Figure 17.

Object No.	1	2	3
RR (%)	99.7	41.0	7.5
Object No.	4	5	6
RR (%)	41.0	15.1	5.2

.

5.3. Estimation of Target Movement

Figure 18 illustrates the experimental result of the estimated target trajectory. The specific scenario is an inclined stair. The red line denotes the target trajectory and the green line denotes the predicted trajectory. This scenario represents a kind of extremal situation for the quadcopter drone during flight. When the target moves back and forth along the stair, the Kalman filter can still accurately predict the moving target trajectory.

Figure 19 displays the real-world experimental results, which shows that even when the target is walking past a shelter, the state estimator can keep tracking the target. In the photo, the red dot denotes the human target center and the green dot shows the predicted point generated by the Kalman filter.

5.4. Dynamic Target Tracking

The test environment includes outdoor grass, indoor corridors, and staircase. Figure 20 demonstrates the outdoor tracking test where the image tracking system, based on the deep learning algorithm proposed, has precisely locked target and started tracking.

As shown in Figure 21, there are two difficulties when the quadcopter is tracking the target under a windy environment. It is difficult for a lightweight quadcopter to maintain a stable altitude because the air pressure sensor might be affected by the environmental disturbance. The second is that the quadcopter itself is susceptible to the influence of wind, which always destabilizes the body attitude. The fuzzy control tuning proposed here has reserved a margin for tracking performance under the severe operating environments.

Figure 22 demonstrates the quadcopter tracking a target who is climbing the stairs. The difficulty in this case is that the target may simultaneously move horizontally and vertically. Thus, the quadcopter speed has to be accurately controlled to avoid the loss of locking.

The case of target tracking in a narrow corridor is shown in Figure 23. This scenario tests the stability of the quadcopter without GPS assistance. For this kind of constricted environment, the quadcopter may suffer from the self-generated wind turbulence while the propelled air rebounds from the wall. Moreover, the indoor sharp corner movement becomes another critical issue to challenge the quadcopter’s capability in space detection and agility of attitude control.

5.5. Feature Comparison

The smart tracking and obstacle avoidance functions of this study are mainly based on the UAV personal security guard system. In order to prove the significance of the research, we cite several documents to show that the functions do not meet the research requirements. The comparison is as follows in

Table 3. Feature comparison table.

Comparator	Application Field	Functional Narrative
This article	Indoor, outdoor (low altitude), stairwell	Identify by clothing; available for crowd, tracking, and obstacle avoidance
[15]	Indoor, outdoor (low altitude)	QR-code identification (strict conditions), tracking but no obstacle avoidance
[23]	Outdoor (high altitude)	Recognition and tracking functions are used for drones to park on the landing zone on the roof of the car
[24]	Outdoor (high altitude)	UAVs identify and track targets at a high altitude
[26]	Indoor	Use 3D vision to measure the distance between UAVs and objects (can be used for obstacle avoidance)
[28]	Outdoor	Using deep learning and simulating multiple UAV tracking and obstacle avoidance (simulation only)

.

6. Conclusions

This paper proposes a lightweight and intelligent UAV tracking and obstacle avoidance system. First, we used the YOLO model with fewer layers to recognize humanoid objects. Secondly, we proposed a lightweight adaptive target identification algorithm, which can extract personal clothing features and filter out complex image backgrounds other than the targets for target identification among crowds. Next, it was necessary to determine the movable space of the UAV, so we proposed the contour and spiral convolutional space detection (CASCSD) algorithm to search the moving area with the shortest distance between the UAV and the object and combine it with the Kalman filter, which estimates the displacement path when obstacles block the tracked object. Finally, a fuzzy adaptive cruise control (ACC) system keeps the drone at an optimal distance while tracking targets. As the selected experimental sites of this study are indoors, outdoors, and in stairwells, it can be applied to the personal security guard scenario.

Because the YOLO model is used for human recognition, it accounts for most of the computing power of the embedded system, resulting in only about ten frames of images during tracking, so the tracking efficiency is not good. However, we have also tested the sum of the other algorithms; it only takes 0.03 s to complete all calculations. We believe that more experts and scholars will propose more lightweight human recognition algorithms in the future. By then, with the lightweight algorithm of this research, the personal security guard system will have higher flexibility and stability.