Automatic Control Optimization for Large-Load Plant-Protection Quadrotor

Abstract

In this article, methods for the attitude control optimization of large-load plant-protection quadrotor unmanned aerial vehicles (UAVs) are presented. Large-load plant-protection quadrotors can be defined as quadrotors equipped with sprayers and a tank containing a large amount of water or pesticide, allowing the quadrotors to water plants or spray pesticide during flight. Compared to the control of common small quadrotors, two main points need to be considered in the control of large-load plant-protection quadrotors—first, the water in the tank gradually diminishes during flight and the physical parameters change during this process. Second, the size and mass of the rotors are especially large, which greatly slows the response rate of the rotors. We present an extended-state reinforcement learning (RL) algorithm to solve these problems. The moment of inertia (MOI) of the three axes and the dynamic response constant of the rotors are included in the state list of the quadrotor during the training process, so that the controller can learn these changes in the models. The controlling laws are automatically generated and optimized, which greatly simplifies the tuning process compared to those of traditional control algorithms. The controller in this article is tested on a 10 kg class large-load plant-protection quadrotor, and the flight performance verifies the effectiveness of our work.

1. Introduction

Quadrotor unmanned aerial vehicles (UAVs) have been widely used in many areas because of their maneuverability and flexibility, as Reference [1] summarized. In plant protection areas, quadrotor UAVs are helping reducing agriculture damage [2]. However, quadrotors are high-dimensional nonlinear underactuated systems with different physical parameters, which increases the difficulty of stable control. Some control algorithms have been tested and applied to quadrotors, and the most widely used algorithm is the traditional proportional-integral-derivative (PID) controller, which achieves an acceptable flight performance. Projects [3,4] are popular open source flight controllers using PID-based control algorithms; the users must tune PID parameters for each quadrotor platform. Another research effort [5] made many improvements to the PID controllers, and the Smith predictor and rotational trajectory planner were augmented. The linear quadratic regulator (LQR) algorithm has also made progress in the control of quadrotor UAVs. One study [6] linearized the quadrotor system and applied the LQR method to quadrotors and energy consumption was also considered. Another report [7] presented a morphing quadrotor, where the arms of the quadrotor could rotate around the body using servos. To achieve stable flight at any configuration, the researchers exploited an LQR-based control strategy that adapts to the drone morphology in real time. However, the performance of these linear control algorithms becomes worse with a larger attitude because the model errors are larger under this condition. Studies [5,7] addressed this problem and compensated for the model errors in their attitude control loops.

Many studies have tested nonlinear control algorithms on quadrotor UAVs. One study [8] achieved stable control of a quadrotor with the backstepping algorithm, and this control method was found to control the quadrotor in any state without nonlinear compensation. Backstepping and sliding mode techniques were applied to an indoor micro quadrotor in another study [9]. A further study [10] developed sliding mode controllers for the attitude loop and the position loop separately. H-infinity control was found to achieve robust control for quadrotors in another study [11], and the effect of external disturbances was minimized. An additional study [12] introduced observers to further compensate for these external disturbances.

The present article focuses on the attitude control of large-load plant-protection quadrotors. Large-load plant-protection quadrotors can be defined as quadrotors equipped with sprayers and a tank containing a large amount of water or pesticide (the mass of water or pesticide can be even heavier than the quadrotors themselves) and the quadrotors can water plants or spray pesticide during flight. Compared to the control of common small quadrotors, some factors need to be considered in the control of large-load plant-protection quadrotors. First, the water in the tank gradually diminishes during application, so the physical parameters change during this process. The mass and moment of inertia (MOI) of the three axes of a plant-protection quadrotor with an empty tank or with a full tank can be very different. The controller must learn these changes because the flight performance of the plant-protection quadrotor can be severely affected. One report [13] presents a fuzzy PID method to solve this problem on plant-protection quadrotor, the PID parameters were adjusted in real time using a fuzzy control table. This problem also appears in the control of the morphing quadrotor in the study [7], and was solved using the adaptive LQR method. Deterministic artificial intelligence (DAI) is a novel intelligent control algorithm that can automatically enforce optimality without any controller tuning. This method has been applied to unmanned underwater vehicles (UUV) [14] and spacecraft [15], and is also able to solve the changing-MOI problem. Second, for large-load plant-protection quadrotors, the size and mass of the rotors are particularly large, which greatly slows the rotor response rate. This factor cannot be ignored; if the response rate is too low, the quadrotor cannot track the desired attitude in a timely manner in any case, which limits the size of electrically powered rotors. Projects [3,4] present feedforward compensation methods to solve this problem. In [5], a Smith predictor is designed to accelerate the response rate of rotors. Another report [7] includes the delay of the rotors in the state equation of the system and solves for the optimal control law using the LQR method. These methods can achieve acceptable performance but rely heavily on parameter tuning, which can be burdensome and dangerous in real applications.

We present a reinforcement learning (RL) method to solve the above problems. RL is an optimization algorithm, where the agents take actions and receive rewards in the environment. In the training process, the agents will optimize the policy to get higher rewards. RL has made remarkable achievements and demonstrated strong potential in complex nonlinear control systems. These techniques have been tested on quadrotor UAVs. One study [16] controlled the full quadrotor using only one neural network in an end-to-end style; the controller tracked the desired position and sent control values to the four rotors directly. In [17], a supplementary RL controller was designed to optimize the performance of a PID-based controller. Another article [18] studied the fault-tolerant control (FTC) problem using the RL method, and a simulation example of a quadrotor was conducted. The report [19] proposes an event-triggered RL method to control the quadrotor, and achieved great performance. However, these RL methods focus on specific platforms and cannot be applied to platforms with other physical parameters or adapt to changes in the physical parameters of the platform. We presented an extended-state RL method in our previous research [20], where the core idea is to extend the critical physical parameters of the system into the state list during the RL training process, and then the controller can automatically fit changes in these parameters. In the present article, we choose the MOIs of three axes and the dynamic response constant of the rotors as critical physical parameters. The RL method in this article is a deterministic policy gradient (DPG) [21] style algorithm. To accelerate the training process, we train 128 quadrotors with different physical parameters in parallel.

Our method automatically generates and optimizes the control law with almost no parameter tuning, which is a great benefit compared to many traditional algorithms. This end-to-end controller design process was found to simplify tuning work and to ensure safety during the experimental process, which is very important for large-load quadrotors. The controller is run on flight controller hardware developed by our team named TensorFly Project that is equipped with a high-performance Raspberry Pi microcomputer, which meets the computational cost of our controller neural network.

The article is organized as follows—in Section 2.1, a nonlinear model of the large-load plant-protection quadrotor system model is established and identification methods are introduced. In Section 2.2, the training process of the extended-state RL algorithm are presented. In Section 3.1–Section 3.4, we introduce the training details and results. Section 3.5 and Section 3.6 present the real flight performance of a 10 kg class large-load plant-protection quadrotor using the RL controller. Section 4 concludes the article.

2. Materials and Methods

2.1. System Model and Identification

In this article, the quadrotor is a 10 kg class large-load plant-protection quadrotor, and Figure 1 introduces the definition of the quadrotor’s frames. In Figure 1a, the attitude of the quadrotor is represented with $\varphi$ (the roll angle), $\phi$ (the pitch angle) and $\psi$ (the yaw angle). In Figure 1b, ${\omega }_1$ to ${\omega }_4$ represent the angular velocities of the four rotors.

The movement and the rotation of the quadrotor is determined by the lift force of the rotors. The dynamic model of the quadrotor is presented in Equation (1):

$$\left\{\begin{array}{c}{J}_{xx}\dot{{\omega }_x}={\omega }_y{\omega }_z\left(J_{yy}-J_{zz}\right)+M_x\hfill \\ J_{yy}\dot{{\omega }_y}={\omega }_x{\omega }_z\left(J_{zz}-J_{xx}\right)+M_y\hfill \\ J_{zz}\dot{{\omega }_z}={\omega }_x{\omega }_y\left(J_{xx}-J_{yy}\right)+M_z,\hfill \end{array}\right.$$

(1)

where ${\omega }_x,{\omega }_y,{\omega }_z$ represent the angular velocities of the three axes and $J_{xx},J_{yy},J_{zz}$ represent the MOIs of the three axes. $M_x,M_y,M_z$ represent the torque generated by the rotors. The torque is generated by the four rotors:

$$\left\{\begin{array}{c}{M}_x=\frac{\sqrt{2}}{2}{C}_{T}d{C}_x\hfill \\ M_y=\frac{\sqrt{2}}{2}{C}_{T}d{C}_y\hfill \\ M_z=C_M{C}_z\hfill \end{array}\right.$$

(2)

$$\left\{\begin{array}{c}{C}_x=-{\omega }_1^2-{\omega }_2^2+{\omega }_3^2+{\omega }_4^2=S_x{a}_x\hfill \\ C_y=-{\omega }_1^2+{\omega }_2^2+{\omega }_3^2-{\omega }_4^2=S_y{a}_y\hfill \\ C_z=-{\omega }_1^2+{\omega }_2^2-{\omega }_3^2+{\omega }_4^2=S_z{a}_z,\hfill \end{array}\right.$$

(3)

where $C_T$ and $C_M$ are the lift coefficient and the torque coefficient of the rotors, and ${\omega }_1$ to ${\omega }_4$ are rotors’ angular velocities. In addition, d represents the arm length of the plant-protection quadrotor; $C_x,C_y,C_z$ and $S_x$, $S_y$, $S_z$ are intermediate variables; and $a_x,a_y,a_z\in \left[-1.0,1.0\right]$ are the control values of the attitude controller. We scale the controller output between $-1.0$ and $1.0$ to simplify the design of controller neural networks. The relationships between the angular velocity of a rotor and the control value of the rotor are modeled in Equation (4):

$$\left\{\begin{array}{c}\omega =\frac{1}{T_{m}s+1}{\omega }_d\hfill \\ {\omega }_d=C_R\sigma ,\hfill \end{array}\right.$$

(4)

where $\omega$ represents the current angular velocity of the rotor and ${\omega }_d$ represents the desired angular velocity of the rotor. $\sigma \in \left[0,1.0\right]$ represents the control value of the rotor, $C_R$ represents the rotor control constant, and $T_m$ represents the dynamic response constant.

System identification is an important aspect in controlling quadrotors and we present several methods to measure or estimate the relevant physical parameters (shown in Figure 2).

Parameters such as the mass and length of arms can be measured directly. MOIs can be measured using the three wire pendulum methods, but these methods require large and expensive equipment for a large-load quadrotor. We estimate the MOIs using the computer aided design (CAD) method; that is, we build an accurate model in SolidWorks software (Figure 2a), weigh every component of the quadrotor, assign the masses of these components in SolidWorks software, and finally automatically calculate the MOIs of the total quadrotor. Moreover, this article focuses on controlling the plant-protection quadrotor with any amount of water carried on it, and the water model must be considered. It is worth noting that, the water in the tank may wobbles during flight, which can greatly worsen the flight performance. To increase the impact of water, we control the quadrotor within a low speed (under 5.0 m/s in the x and y axes, under 1.0 m/s in the z axis) and a low accelerated velocity (under 1.0 m/s${}^2$ in the three axes). Moreover, the tank in this project is an anti-oscillation tank specially designed for plant-protection quadrotor, where water is separated in small blocks, which greatly eliminate the impact of water. As a result, we assume that the water is shaped as a cuboid, as shown in Figure 3.

In Figure 3, $co{g}_{bo}$ is the center of gravity (COG) of the body (including the empty tank), $co{g}_{wa}$ is the COG of the water cube, and $co{g}_{all}$ is the COG of the total quadrotor. In addition, $d_{cog}$ denotes the distance between $co{g}_{bo}$ and $co{g}_{wa}$, $d_{bo}$ represents the distance between $co{g}_{bo}$ and $co{g}_{all}$, and $d_{wa}$ is the distance between $co{g}_{wa}$ and $co{g}_{all}$. To simplify the calculation, the movement of the COG of the water is ignored, so $d_{cog}$ is constant. $V_{wa}$ is the volume of the water cube, and $l_{wa}$ is the length and width of the water cube, so the height of the water cube can be calculated as $h_{wa}=V_{wa}/l_{wa}^2$. Then, $d_{bo}$ and $d_{wa}$ can be calculated as Equation (5) presents:

$$\left\{\begin{array}{c}{d}_{bo}=\frac{V_{wa}{\rho }_{wa}}{m_{bo}+V_{wa}{\rho }_{wa}}{d}_{cog}\hfill \\ d_{wa}=\frac{m_{bo}}{m_{bo}+V_{wa}{\rho }_{wa}}{d}_{cog},\hfill \end{array}\right.$$

(5)

where ${\rho }_{wa}$ is the density of the water or pesticide. The MOIs of the water can be calculated using Equation (6):

$$\left\{\begin{array}{c}{J}_{xx,wa}=\frac{V_{wa}{\rho }_{wa}}{12}\left(l_{wa}^2+h_{wa}^2\right)\hfill \\ J_{yy,wa}=\frac{V_{wa}{\rho }_{wa}}{12}\left(l_{wa}^2+h_{wa}^2\right)\hfill \\ J_{zz,wa}=\frac{V_{wa}{\rho }_{wa}}{6}{l}_{wa}^2,\hfill \end{array}\right.$$

(6)

where $J_{xx,wa}$, $J_{yy,wa}$, and $J_{zz,wa}$ are the MOIs of the water cube in the three axes. Then, the MOIs of the total quadrotor can be calculated using rigid-body parallel axis theory, as Equation (7) presents:

$$\left\{\begin{array}{c}{J}_{xx}=J_{xx,bo}+J_{xx,wa}+V_{wa}{\rho }_{wa}{d}_{wa}^2+m_{bo}{d}_{bo}^2\hfill \\ J_{yy}=J_{yy,bo}+J_{yy,wa}+V_{wa}{\rho }_{wa}{d}_{cog}^2+m_{bo}{d}_{bo}^2\hfill \\ J_{zz}=J_{zz,bo}+J_{zz,wa},\hfill \end{array}\right.$$

(7)

where $J_{xx,bo}$, $J_{yy,bo}$, and $J_{zz,bo}$ are the MOIs of the quadrotor body in the three axes. In this way, we can estimate the MOIs of the total quadrotor with any amount of water carried in it.

In this article, we identify the parameters of the motors and rotors with a rotor bench (Figure 2b). The bench can measure the lift force, the torque and the angular velocity of the rotor, so the lift coefficient $C_T$ and torque coefficient $C_M$ can be calculated. The rest of the unknown parameters, such as the dynamic response constant of the rotors $T_m$, can be found in an online quadrotor database: https://flyeval.com/ (accessed on 27 March 2021) [22].

In the simulation and training process, the simulation environment updates the states of the quadrotors, including the attitude, the angular velocities of the three axes and the angular velocities of the rotors. The attitude is described in unit quaternions in this article:

$$\left\{\begin{array}{c}{q}_0=cos\left(\frac{\Theta }{2}\right)\hfill \\ q_1=sin\left(\frac{\Theta }{2}\right)p_x\hfill \\ q_2=sin\left(\frac{\Theta }{2}\right)p_y\hfill \\ q_3=sin\left(\frac{\Theta }{2}\right)p_z,\hfill \end{array}\right.$$

(8)

where $\Theta$ is the combined rotation of the roll angle, the pitch angle and the yaw angle, and $p_x,p_y,p_z$ are the projections of $\Theta$ in the roll, pitch and yaw axes, respectively. The quaternions can be updated with angular velocities in three axes using the Runge-Kutta method:

$$\left[\begin{array}{c}\dot{q_0}\\ \dot{q_1}\\ \dot{q_2}\\ \dot{q_3}\end{array}\right]=\frac{1}{2}\left[\begin{array}{cccc}0& -{\omega }_x& -{\omega }_y& -{\omega }_z\\ {\omega }_x& 0& {\omega }_z& -{\omega }_y\\ {\omega }_y& -{\omega }_z& 0& {\omega }_x\\ {\omega }_z& {\omega }_y& -{\omega }_x& 0\end{array}\right]\left[\begin{array}{c}{q}_0\\ q_1\\ q_2\\ q_3.\end{array}\right]$$

(9)

2.2. Attitude Controller Design for Large-Load Plant-Protection Quadrotors

The attitude controller receives the desired attitude $at{t}_d$, the current attitude $att$, the angular velocities in the three axes ${\omega }_x$,${\omega }_y$,${\omega }_z$, the MOIs in the three axes $J_{xx},J_{yy},J_{zz}$ and the response constant of the rotors $T_m$, and then outputs the control values of the three axes $a_x,a_y,a_z\in \left[-1.0,1.0\right]$ according to the states. The control allocator transfers $a_x,a_y,a_z$ and the desired throttle value $t{h}_d$ to the control values of each rotor ${\sigma }_i,i=1,2,3,4$. The control diagram is presented in Figure 4.

2.2.1. Quadrotor Attitude Controller with Reinforcement Learning

The RL frame includes the following factors—the agent, environment, state ${\mathit{s}}_t$, action ${\mathit{a}}_t$, policy $\pi \left({\mathit{s}}_t\right)$ and reward $r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)$. During the training process, the environment outputs the state ${\mathit{s}}_t$, and the agent takes action ${\mathit{a}}_t$ with the policy $\pi \left({\mathit{s}}_t\right)$ and then obtains a reward $r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)$. To describe the performance of the agent in the long term, the value of the current state and action $V_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)$ is introduced:

$$V_t=r_t+\lambda r_{t+1}+{\lambda }^2{r}_{t+1}+...=r_t+\lambda V_{t+1},$$

(10)

where $\lambda \in \left[0,1\right)$ is the discount factor. Equation (10) shows that the stat-action value $V_t$ is the sum of expected future rewards. Although these rewards are unknown, we can estimate them from previous outcomes. We can assume that an optimal policy ${\pi }_{opt}$ can maximize $V_{t+1}$:

$$\begin{array}{cc}\hfill V_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)& =r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)+\lambda ma{x}_{{\mathit{a}}_t+1\in A}\left(V_{t+1}\left({\mathit{s}}_{t+1},{\mathit{a}}_{t+1}\right)\right)\hfill \\ \hfill & =r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)+\lambda V_{t+1}\left({\mathit{s}}_{t+1},{\pi }_{opt}\left({\mathit{s}}_{t+1}\right)\right),\hfill \end{array}$$

(11)

where A is the action space of the agent. During the training process, we apply the current policy $\pi$ instead of ${\pi }_{opt}$.

In this article, the agent is a quadrotor with four rotors, and the environment receives the angular velocities of the rotors and outputs the attitude and angular velocities in three axes.

2.2.2. Network Structure and Training Algorithm

The structure of the RL controller is an actor-critic [21] based structure, as presented in Figure 5. The critic network is bulit to estimate the value of the state and action, and the actor network is bulit to compute the control values with the state.

The RL method in this article is the extended-state RL proposed in our previous work [20], where system parameters can be extended into the state list, which enables the controller to fit the changes in these parameters. As a result, the state list consists of the attitude errors $at{t}_e=\left[q_0,q_1,q_2,q_3\right]$ and the current angular velocities in the three axes $\left[{\omega }_x,{\omega }_y,{\omega }_z\right]$, along with the MOIs in the three axes $J_{xx},J_{yy},J_{zz}$ and the response constant of the rotors $T_m$. The action list consists of control values in the three axes $a_x,a_y,a_z$.

The training method in this article is the DPG method [21], consists of two actor networks and two critic networks. The actor networks include the target actor network and the online actor network, both with the same structure, and two hidden layers of 128 nodes are included. The tanh activation function is set to limit the output value between $-1.0$ and $1.0$ in the output layer. The online actor network is trained in each time step, the gradients are computed with the sampled policy gradients, and the weights are updated using the Adam optimizer [23]. The target actor network is soft updated [21] using the online actor network.

The critic networks also include the target critic network and the online critic network. The two critic networks share the same structure as well, and two hidden layers of 128 nodes are designed. The cost function of the online critic network is a squared error function, and we lower the cost with the Adam optimizer. We soft update the weights of the target critic network using the online critic network.

The reward function design is critical work in the RL algorithm. Squared error style reward functions are widely used in the RL-based control domain, as Equation (12) presents:

$$r\left(\mathit{s}\right)=-p_1*{\Theta }^2-p_2*\left({\omega }_x^2+{\omega }_y^2+{\omega }_z^2\right),$$

(12)

where $p_1$ and $p_2$ are positive parameters and $\Theta =2arccos\left(q_0\right)$ is the total rotation angle of the attitude error. This reward function encourages the agent to stably reach the zero point. However, in many RL studies using these squared error reward functions, controllers show obvious steady-state errors [24]. We analyze the cause of steady-state errors and present a reward shaping method to solve this problem in article [20]. The reward shaping skill is also applied in this article, and we construct a piecewise function as:

$$fr\left(x,ϵ,\varrho \right)=\left\{\begin{array}{cc}-\left|x\right|/ϵ+1,\hfill & \left(\left|x\right|\le ϵ\right)\hfill \\ \left(\left|x\right|-ϵ\right)/\left(ϵ-\varrho \right),\hfill & \left(ϵ<\left|x\right|<\varrho \right),\hfill \end{array}\right.$$

(13)

where x is the input state value (attitude error or angular velocity in this article), $ϵ$ and $\varrho$ are constants. With Equation (13), the reward function is designed as:

$$r\left(\mathit{s}\right)=p_1*fr\left(\Theta ,{\delta }_1,{\rho }_1\right)+p_2*\left(fr\left({\omega }_x,{\delta }_2,{\rho }_2\right)+fr\left({\omega }_y,{\delta }_2,{\rho }_2\right)+fr\left({\omega }_z,{\delta }_2,{\rho }_2\right)\right),$$

(14)

where $p_1,p_2,{\delta }_1,{\rho }_1,{\delta }_2$ and ${\rho }_2$ are positive parameters to be tuned. Compared to the reward functions like Equation (12), this function output positive reward for close-to-zero states.

In a real flight environment, sensor noise can decrease the performance of the controller. The attitude angle data are estimated and filtered using gyroscope data, accelerometer data and magnetic compass data, so the errors of attitude angle data cannot be ignored. The angular velocities from sensors are relatively accurate, so errors of angular velocity data are ignored in this research. To improve the the robustness for sensor noise of the quadrotor, sensor noise are added to the agents in the training process.

In our algorithm, we run several agents in parallel and immediately employ their experience at each time step, which has some advantages:

(1)

Different agents have different physical parameters with different initial states and take different actions, thus avoiding the experience correlation problem.

(2)

Compared to traditional RL methods such as deep Q learning (DQN) [25], parallel training does not need experience replay skills and thus requires much less memory.

(3)

Several agents can provide more experience at the same time, greatly accelerating the training process.

The whole algorithm is listed as Algorithm 1.

Algorithm 1: Extended-state RL parallel training algorithm

$n_{agent}$: max number of agents

$e_{max}$: max number of episodes

$s_{max}$: max number of steps during an episode

$R\left({\mathit{s}}^{\prime }\right)$: The reward function

$\tau$: soft update rate

1: create $n_{agent}$ agents with different system parameters

2: randomly initialize the weights of the online critic network ${\mathbf{\theta }}_v$ and the online actor network ${\mathbf{\theta }}_{\pi }$

3: initialize the weights of the target critic network ${\mathbf{\theta }}_v^{\prime }$ and the target actor network ${\mathbf{\theta }}_{\pi }^{\prime }:{\mathbf{\theta }}_v^{\prime }\leftarrow {\mathbf{\theta }}_v,{\mathbf{\theta }}_{\pi }^{\prime }\leftarrow {\mathbf{\theta }}_{\pi }$

4: for  $epi=1,2\dots e_{max}$do

5:   initialize all agents with random states

6:   for $step=1,2\dots s_{max}$ do

7:     clear the memory box $B\leftarrow ⌀$

8:     for $n=1,2\dots n_{agent}$ do

9:      recieve the current state $\mathit{s}$

10:      take action $\mathit{a}$ according to $\mathit{s}$

11:      receive the next state ${\mathit{s}}^{\prime }$

12:      put $E=\left(\mathit{s},\mathit{a},{\mathit{s}}^{\prime }\right)$ into the memory box $B\leftarrow B\bigcup E$

13:     end for

14:     for $i=1,2\dots n_{agent}$ do

15:      obtain experience ${\mathbf{e}}_i=\left({\mathit{s}}_i,{\mathit{a}}_i,{\mathit{s}}_i^{\prime }\right)$ from B

16:      compute the reward of experience ${\mathbf{e}}_i:r_i\leftarrow R\left({\mathit{s}}_i^{\prime }\right)$

17:      $y_i\leftarrow r_i+\lambda V\left(\left({\mathit{s}}_i^{\prime },\pi \left({\mathit{s}}_i^{\prime }|{\mathbf{\theta }}_{\pi }\right)\right)|{\mathbf{\theta }}_v\right)$

18:      update ${\mathbf{\theta }}_v$ by minimizing the loss: $L=\frac{1}{N}{\sum }_i{\left(y_i-V\left({\mathit{s}}_i,{\mathit{a}}_i|{\mathbf{\theta }}_v\right)\right)}^2$

19:      update ${\mathbf{\theta }}_{\pi }$ using the sampled policy gradients:

${▽}_{{\mathbf{\theta }}_{\pi }}\approx \frac{1}{N}{\sum }_i{▽}_{\mathit{a}}V\left(\left(\mathit{s},\mathit{a}|{\mathbf{\theta }}_v\right){|}_{\mathit{s}={\mathit{s}}_i,\mathit{a}={\mathit{a}}_i}{▽}_{{\mathbf{\theta }}_{\pi }}\pi \left(s|{\mathbf{\theta }}_{\pi }\right)\right){|}_{\mathit{s}={\mathit{s}}_i}$

20:      soft update ${\theta }_v^{\prime }$ with ${\mathbf{\theta }}_v:{\mathbf{\theta }}_v^{\prime }\leftarrow \tau {\mathbf{\theta }}_v+\left(1-\tau \right){\mathbf{\theta }}_v^{\prime }$

21:      soft update ${\theta }_{\pi }^{\prime }$ with ${\mathbf{\theta }}_{\pi }:{\mathbf{\theta }}_{\pi }^{\prime }\leftarrow \tau {\mathbf{\theta }}_{\pi }+\left(1-\tau \right){\mathbf{\theta }}_{\pi }^{\prime }$

22:     end for

23:   end for

24: end for

3. Results

3.1. Training Environment

A simulation environment is developed to train the controller using Python 3.8 and TensorFlow 1.14. We can create quadrotors with desired physical parameters and test the controllers in it. The time steps are not associated with real time, so the training process can be much faster than in the real world. The states of the quadrotors do not update in each time step until the training or control process is finished. We train the controller using a laptop computer equipped with a i7-7700HQ central processing unit (CPU) and an NVIDIA GTX1060 graphics processing unit (GPU).

3.2. Quadrotor Details

The large-load plant-protection quadrotor is a 10 kg class “X” type plant-protection quadrotor equipped with HLY Q9L brushless motors, 3080 carbon fiber propellers, Hobbywing XRotor Pro HV 80A electronic speed controls (ESCs) and two ACE TATTU 16,000 mAh 6S Li-polymer batteries in series. The parameters of the quadrotor with an empty tank are identified in Section 2.1 and listed in

Table 1. Physical parameters of the quadrotor with an empty tank.

Parameter	Value
Mass m	8 kg
Moment of inertia $J_{xx}$	$1.066$ kg·m${}^2$
Moment of inertia $J_{yy}$	$0.961$ kg·m${}^2$
Moment of inertia $J_{zz}$	$1.843$ kg·m${}^2$
Arm length d	$0.65$ m
Lift coefficient $C_T$	$5.7894$×${10}^{-4}$ N/(rad/s)${}^2$
Torque coefficient $C_M$	$2.283\times {10}^{-5}$ N·m/(rad/s)${}^2$
Rotor control constant $C_R$	316 rad/s
Rotor dynamic response constant $T_m$	$0.035$ s
Volume of the tank $V_{tank,max}$	10 L
Maximum mass of water $M_{wa,max}$	10 kg
Density of water ${\rho }_{wa}$	$1.0\times {10}^3$ kg/m${}^3$
Distance between the quadrotor and water $d_{cog}$	$0.3$ m
Length and width of the water cube $l_{wa}$	$0.28$ m

. The water in the tank is idealized as a cube with equal length and width.

3.3. Training Details

The rotor dynamic response constant of this large-load quadrotor is $T_m=0.035$ s, and we calculate that this constant for a common small quadrotor is $T_m=0.0157$ s. During the training process, $T_m$ is generated randomly between 0.015 s and 0.045 s for agents.

Using the data in Table 1, we can calculate the maximum and minimum MOIs in three axes using Equation (7), and the results are listed in

Table 2. Range of MOIs in three axes.

Parameter	Value
Maximum MOI in the x-axis	$1.540$ kg·m${}^2$
Minimum MOI in the x-axis	$1.066$ kg·m${}^2$
Maximum MOI in the y-axis	$1.435$ kg·m${}^2$
Minimum MOI in the y-axis	$0.961$ kg·m${}^2$
Maximum MOI in the z-axis	$1.973$ kg·m${}^2$
Minimum MOI in the z-axis	$1.843$ kg·m${}^2$

:

As a result, the MOIs in the x and y axes range between $0.8$ and $1.7$ kg·m${}^2$, and the MOI in the z axes ranges between $1.8$ and $2.1$ kg·m${}^2$ during the training process.

In this research, sensor noise is added during the training process to improve the robustness of the controller. The attitude and heading reference system (AHRS) used in our flight controller is a mature product, so the attitude angle data and angular velocity data can be read directly from the module. The AHRS module claims that the errors of attitude angle in the roll and pitch axes are between $\pm 0.1^{\circ }$ and the errors of attitude angle in the yaw axis are between $\pm 1^{\circ }$. As a result, we add noise conforming to uniform distributions between $\pm 1^{\circ }$ to attitude angle data in the three axes during the training process.

Table 3. Training hyperparameters.

Parameter	Value
Discount factor $\gamma$	$0.1$
Learning rate of the critic net ${\alpha }_{critic}$	$0.001$
Learning rate of the actor net ${\alpha }_{actor}$	$0.001$
Batch size N	128
Soft update rate $\tau$	$0.01$
Maximum number of agents $n_{agent}$	128
Maximum number of training episodes $e_{max}$	10,000
Control time step $DT$	$0.01$ s
State update time step $dt$	$0.0001$ s
Maximum time step in an episode $t_{max}$	5 s
Reward function constant $p_1$	$1.0$
Reward function constant ${ϵ}_1$	$0.01$
Reward function constant ${\varrho }_1$	$\pi$
Reward function constant $p_2$	$0.1$
Reward function constant ${ϵ}_2$	$0.1$
Reward function constant ${\varrho }_2$	$2\pi$

presents the training hyperparameters. The total training process takes approximately 3 h.

3.4. Performance in Simulation

The RL controller in this article can generate an optimal control law with the MOIs in three axes and the rotor dynamic response constant input. We first test the performance of the plant-protection quadrotor without water in the tank, that is: $J_{xx}=1.066$, $J_{yy}=0.961$, $J_{zz}=1.843$, and $T_m=0.035$. The performance along the roll axis and yaw axis are shown in Figure 6 and Figure 7, respectively.

The convergence rate of our controller is very high, and nearly no overshoot or steady-state error exists. Next, we test the performance of plant-protection quadrotors with a half-full water tank ($J_{xx}=1.377$, $J_{yy}=1.272$, $J_{zz}=1.908$, and $T_m=0.035$) and a full water tank ($J_{xx}=1.540$, $J_{yy}=1.435$, $J_{zz}=1.973$, $T_m=0.035$). The performance along the roll axis is given in Figure 8. We conclude that our controller can adapt to changes in the system parameters and keep the quadrotor flying stably and accurately with any amount of water in the tank.

We test the robustness for sensor noise of the controller in Figure 9. Noises conforming to uniform distributions between $\pm 1^{\circ }$ are added to attitude angle data of the three axes, the proposed controller still keeps the attitude of the quadrotor around the desired attitude angle, and the attitude errors are less than $1^{\circ }$, which is acceptable.

Traditional PID controllers are designed for comparison, and their performance and the performance of our RL controller are presented below. As show in Figure 10, PID controller 1 performs well for the quadrotor without water in the tank, but when the tank is filled with water, PID controller 1 shows obvious overshoot compared to our RL controller. As show in Figure 11, PID controller 2 performs well for the quadrotor with a full water tank, but when the tank is empty, PID controller 2 shows a notably lower convergence rate than our RL controller. In conclusion, it is difficult to control the quadrotor well with any amount of water using the same PID parameters, and the performance of our RL controller is much more robust than that of traditional PID controllers with system parameter changes.

In the training process, the states of the quadrotors are initialized randomly, so the quadrotor can recover from any state (even the full turnover state) to zero points. During the training process, the performance of our controller improves gradually. We designed tests to verify these two factors. We test controllers in different episodes (ep = 0, 1000, 5000 and 10,000) on the roll axis of the quadrotor without water in the tank. In Figure 12, the quadrotors recovered to zero point states from random initial states 100 times. We conclude that, after the training process, the controller tends to recover from any state to the zero point and that the performance improves as the training proceeds.

3.5. Flight Controller Platform Details

The flight controller in this project is developed by our team and is called the TensorFly Project; this controller focuses on flight control with RL methods. To meet the demand of real time network calculation, we choose a Raspberry Pi 3B microcomputer (four 1.2 GHz CPU cores) as the main controller in our project. The controller is coded with C++ and calculates matrixes using the eigenlibrary [26]. The flight controller runs the Raspbian operating system on the board, and we perform substantial work toward reducing the latency of the system. The actor network is calculated using the eigenlibrary [26] and, on average, the computation process takes approximately 260 $\mathsf{\mu }s$. However, in rare cases, the computation process may take over 1 ms and we think the problem may result from background processes of the onboard operating system. Regardless, the time consumption of the network calculation process is far less than the control period (10 ms), which is acceptable. The AHRS module contained in this controller is a JY901 module, which outputs attitude angle data and angular velocity data in real time directly, and the attitude errors are less than $1.0^{\circ }$. The flight controller and the AHRS module are shown in Figure 13.

3.6. Flight Performance in Real Flight

The 10 kg class large-load plant-protection quadrotor and its flight using our RL controller are presented in Figure 14.

We test the flight performance of the attitude tracking task on this quadrotor without water and with a half-full water tank, and the performance is presented below. We control the desired attitudes of the quadrotor using a remote controller manually. The flight log of the quadrotor without water in the tank ($J_{xx}=1.3660$, $J_{yy}=1.261$, $J_{zz}=2.343$, $T_m=0.035$) is presented in Figure 15, and the attitude errors are presented in Figure 16. The flight log of the quadrotor with a half-full water tank ($J_{xx}=1.3660$, $J_{yy}=1.261$, $J_{zz}=2.343$, $T_m=0.035$) is presented in Figure 17, and the attitude errors are presented in Figure 18.

We conclude that the quadrotor can rapidly and smoothly track the desired attitude (the attitude errors are within $6^{\circ }$ during flight), regardless of the amount of water in the tank, which is very practical in real applications.

4. Conclusions

4.1. Discussion

In this article, we present an extended-state RL method for large-load plant-protection quadrotors. We focus on critical problems of plant-protection quadrotors in real applications, such as changes in the moment of inertia during watering work and the slow response rates of large rotors. The RL controller extends these parameters in the state list during training and running, so the controller can automatically adapt to these changes. The introduction of excellent training skills greatly accelerates the training process, and reward shaping skills eliminate steady-state errors. The controller is tested in a simulation environment and applied on a real large-load plant-protection quadrotor platform and the controller achieves great performance regardless of the amounts of water in the tank. Compared to traditional methods that need manual modification and tuning of the controller, our methods show an end-to-end design style, greatly simplify the controller design work and ensure safety in real flight experiments.

4.2. Future Work

During our experimental process, we found that uncertainties such as side winds, movement of the COG of the quadrotor, wobbling water in the tank and external disturbances can increase the flight stability. Active disturbance rejection control (ADRC) methods will be applied and tested in our future work to solve these practical problems for large-load plant-protection quadrotors.

The extended-state RL method proposed in this paper can be widely used on several other platforms such as unmanned underwater vehicles and robot arms. We note that the DAI methods can achieve a similar performance, and we plan to consider this comparative methodology to further improve the performance of our methods.