Single-Parameter-Tuned Attitude Control for Quadrotor with Unknown Disturbance

Abstract

In this paper, methods are presented for designing a quadrotor attitude control system with disturbance rejection ability, wherein only one parameter needs to be tuned for each axis. The core difference between quadrotor platforms are extracted as critical gain parameters (CGPs). Reinforcement learning (RL) technology is introduced in order to automatically optimize the controlling law for quadrotors with different CGPs, and the CGPs are used to extend the RL state list. A deterministic policy gradient (DPG) algorithm that is based on an actor-critic structure in a model-free style is used as the learning algorithm. Mirror sampling and reward shaping methods are designed in order to eliminate the steady-state errors of the RL controller and accelerate the training process. Active disturbance rejection control (ADRC) is applied to reject unknown external disturbances. A set of extended state observers (ESOs) is designed to estimate the total disturbance to the roll and pitch axes. The covariance matrix adaptation evolution strategy (CMA-ES) algorithm is used to automatically tune the ESO parameters and improve the final performance. The complete controller is tested on an F550 quadrotor in both simulation and real flight environments. The quadrotor can hover and move around stably and accurately in the air, even with a severe disturbance.

1. Introduction

Quadrotor unmanned aerial vehicles (UAVs) have shown great potential for both civil and military applications, because of low cost, small volumes, simple structures, flexible operation, and vertical take-off and landing capability [1,2]. However, quadrotor UAVs present control challenges because of high-dimensional nonlinear dynamics, difficulties in measuring system parameters, and ever-changing noise and disturbances. Various control algorithms have been tested on quadrotor UAVs, among which the classic proportional-integral-derivative (PID) controller is the most widely used. For example, paper [3] presents a quadrotor UAV attitude control method to realize complex acrobatic UAV maneuvers using the double-ring PID algorithm and smith predictor. In paper [4], quadrotor attitude and position controller using rotation matrix based PID algorithm are designed. The linear quadratic regulator (LQR) algorithm has also being used in quadrotor UAVs with some success [5]. Researchers compared the performance of a Qball-X4 quadrotor with PID controller and LQR controller, and the tracking trajectory with LQR controller is more smooth and it has less disturbance [6]. LQR algorithm can also be applied on a morphing quadrotor and achieve a success [7]. Nonlinear control algorithms, such as back-stepping (BS), sliding mode control, and H-infinity control, have been applied to quadrotors. Researchers propose an approach of the backstepping control running parallel with a sliding mode observer for a quadrotor UAV [8,9]. Paper [10] presents quadrotor altitude control with sliding mode control and paper [11] designed attitude and position control using sliding mode control. H-infinity control is another optimal control algorithm that provides a robust stable closed-loop system with high performance tracking and disturbance rejection. Paper [12] present H-infinity style quadrotor attitude controller, where the system transfer function is identified based on input and output experimental data. In paper [13], a H-infinity observer is designed to compensate the effect of actuator failures, sensor failures, and external disturbances. However, these algorithms rely heavily on model accuracy, which severely affects application to real quadrotor platforms.

The active disturbance rejection controller (ADRC) technique is based on the traditional PID: a tracking differentiator (TD) is set to arrange the transition process and extract the differential signals, and an extended state observer (ESO) is used to estimate the state of the control object and the unknown disturbances. A nonlinear state error feedback (NLSEF) is formed by a nonlinear combination of the results from the TD and the ESO, including the compensation for external disturbances if needed [14]. Both external disturbances and model errors can be estimated while using ESOs. The critical gain parameter (CGP) is the gain in the input channel that characterizes the proportional relation between the input signal and the derivative of the process output of a specific order [15]. In this paper, the CGP is the proportional relation between the control signal and the angular accelerations of three axes. Studies have shown that the ADRC can be used to stably control a quadrotor in the presence of external disturbances [16,17,18].

Acceptable control performance have been demonstrated for the control systems proposed by previous researchers. However, model-based control algorithms, such as LQR and BS, heavily rely on model accuracy, which hinders their application to real quadrotor platforms. It is difficult to tune the parameters of model-free algorithms, such as PID and ADRC. When considering the traditional double-ring PID algorithm as an example, typically three or more parameters must be tuned for each axis, and the tuning process can be time consuming and dangerous. For quadrotor systems, automatic tuning algorithms, such as the genetic algorithm (GA) and particle swarm optimization (PSO), can be tested in simulation environments, but it is unsafe to automatically explore control parameters during real flight experiments. In this paper, we present an Reinforcement learning (RL) method to achieve single-parameter-tuned attitude control. Thus, the difficulty of multi-parameter tuning in PID-based algorithms is reduced.

RL is an automated self-optimization method that allows for agents to explore the environment and collect experience to improve performance. Agents output actions and receive rewards during the exploration process, and the controller, referred to as the policy in this frame, constantly optimizes itself based on experience to obtain higher rewards. The process can present optimal or at least locally optimal controllers, which is sufficient for solving a complex control problem in many cases. RL algorithms have been introduced into the UAV control area with remarkable achievements [19,20]. Andrew Ng et al. used the RL method to design a flight controller for the Stanford autonomous helicopter [21]. A data-driven approach was used to determine the system model, which is, flight data were collected and fitted using a locally weighted linear regression algorithm. These approaches are effective for some specific experimental platforms, but cannot be directly transferred to other types of platforms, and measuring of system parameters or identifying system models can be difficult and expensive. An alternative approach is to train a policy that is sufficiently robust in order to neglect model errors as well as potential disturbances. For example, computer-aided design (CAD) has been used to obtain the parameters of a quadrupedal robot in [22], and noise and disturbances can be added to either the system models or sensor data during the training process. We tested this method on a quadrotor in our previous study, and the trained policy was effective for platforms with different system parameters in a simulation environment. However, the performance for each platform may be less ideal than a full tuned PID controller.

In this paper, a new method is presented in order to achieve simple tuning control with RL: that is, we extract the core difference of platforms as critical system parameters that are then included in the state list during the RL training process. This method can be used to apply the trained controller to different platforms, and excellent performance can be achieved with simple parameter tuning. The critical system parameters for quadrotor UAV systems include the moment of inertia, the lift coefficient, the torque coefficient, etc. We extract these parameters as CGPs for the three axes, which is a key step in the ADRC algorithm. This method can be used to apply the RL controller to different quadrotors, where only one parameter CGP needs to be tuned for each axis.

In many previous studies, RL controllers have exhibited excellent performance, including a fast convergence rate with hardly any overshoot, but they have failed to eliminate steady-state errors. These errors remain distinct, irrespective of how the hyperparameters are set and the types of exploration strategies applied [23]. In [23], integral compensators are added to the quadrotor system to eliminate steady-state errors, but our simulation experimental results show that this method may reduce the final performance. Thus, we develop novel methods, i.e., reward function shaping and mirror sampling, to solve the steady-state error problem that is incurred using RL algorithms. Ideal results with nearly no steady-state error are obtained, and the training process is considerably accelerated.

External disturbances to a quadrotor are compensated for by designing a set of ESOs to estimate the total disturbance of the roll and pitch axes, where the total controller output is the combination of the RL controller output and the ESO feedback. However, it is difficult to tune the parameters of an ESO. Inappropriate ESO parameters cause high frequency vibration problems in both simulation and real flight experiments, which may reduce the control quality. In this paper, an efficient and powerful metaheuristic algorithm, which is referred to as the covariance matrix adaptation evolution strategy (CMA-ES), is introduced to automatically adjust the ESO parameters. The CMA-ES is a type of random numerical derivative-free optimization algorithm with a superior efficiency and success rate to GA and PSO [24].

In this paper, the CMA-ES algorithm is designed to reduce the error between the ESO output and the real disturbance. Disturbance measurement is challenging. We meet this challenge by designing a specialized test: the quadrotor is placed on the ground in the absence of an external disturbance, and control signals are sent to the controller; however, the motors are not actuated and they remain still during the test. In this situation, from the perspective of the controller, a disturbance compensates for the control signals to keep the quadrotor still, such that the value of the disturbance is simply the negative of the current control signals multiplied by the CGP. This test prevents problems, such as safety risks, computational costs, and battery capacity limitation, and we can run the CMA-ES algorithm directly on a real quadrotor.

In this paper, the quadrotor attitude control is divided into three second-order linear systems with different CGPs and disturbances. We lock the pitch and yaw axis of a quadrotor as a standard second-order linear rotation system rather than using a full quadrotor during RL training process. Our RL algorithm is based on the deterministic policy gradient (DPG) algorithm with two actor neural networks (NNs) and two critic NNs. Mirror sampling and reward shaping methods are introduced to the training process to eliminate the steady-state errors of the RL controller and accelerate the training process. The CGPs of the quadrotor platforms are recognized while using TDs, and the CGPs can also be manually tuned in a real flight environment. The CMA-ES algorithm is used to automatically conduct ESO parameter tuning in both a simulation environment and a real quadrotor. The controllers are well-trained and tuned and they are also tested in a quadrotor simulation system. Finally, the controller is applied to a real F550 quadrotor. The program is run on a Raspberry Pi microcomputer, which is sufficiently fast to run the NNs in real time. The quadrotor can hover and move around stably and accurately in the air even under severe disturbances.

The main contributions of this paper are summarized below:

(1)

In this paper, RL is used to develop a single-parameter-tuned quadrotor attitude control system that exhibits excellent performance in both simulation and real flight experiments. This method offers the considerable advantage of requiring one parameter CGP to be tuned for each axis. Critical system parameters are used to extend the RL state list, making the same RL network suitable for platforms with different system parameters, which represents an advance in RL theory.

(2)

The introduction of reward function shaping and mirror sampling eliminate the steady-state errors incurred using RL control systems and considerably accelerate the training process.

(3)

ADRC methods are used to eliminate external disturbances to the quadrotor attitude control system in real time. A practical method is developed to automatically tune the ESO parameters using CMA-ES for a real quadrotor.

The remainder of this paper is organized, as follows. In Section 2, a nonlinear quadrotor system model is established, and identification methods are introduced. A RL control algorithm and a CMA-ES parameter tuning algorithm are developed in Section 3. In Section 4, the training details and results are presented, and the simulation experiments are discussed. Section 5 presents the ESO tuning details for a real quadrotor platform and the performance of real flight experiments using the RL controller. Section 6 concludes the paper.

2. System Model and Identification

In this paper, an “X” type F550 quadrotor is used and tested, and its structure and frame are presented in Figure 1. The attitude of the quadrotor represents the angular error between the body frame and the earth frame.In Figure 1, $\varphi ,\phi ,\psi$ represent the roll, pitch, and yaw angles, respectively, while ${\omega }_1$ to ${\omega }_4$ represent angular velocities of four rotors.

The rotation of a quadrotor can only be controlled by changing the lift force and torque of the four rotors. The rigid-body dynamic model of the quadrotor in this paper can be derived from the Newton–Euler equation:

$$\left\{\begin{array}{c}{J}_{xx}\dot{{\omega }_x}={\omega }_y{\omega }_z\left(J_{yy}-J_{zz}\right)+M_x+D_x\hfill \\ J_{yy}\dot{{\omega }_y}={\omega }_x{\omega }_z\left(J_{zz}-J_{xx}\right)+M_y+D_y\hfill \\ J_{zz}\dot{{\omega }_z}={\omega }_x{\omega }_y\left(J_{xx}-J_{yy}\right)+M_z+D_z\hfill \end{array}\right.$$

(1)

where ${\omega }_x,{\omega }_y,{\omega }_z$ are angular velocities of the quadrotor. $J_{xx},J_{yy},J_{zz}$ are the moment of inertia in three axes. $M_x,M_y,M_z$ are torque generated by rotors. $D_x,D_y,D_z$ are disturbance torque in three axes, including aerodynamic drag, side wind, center of gravity shift, physical impact, and model error. The torque of three axes are determined by the angular velocities of four rotors:

$$\left\{\begin{array}{c}{M}_x=\frac{\sqrt{2}}{2}{C}_T\ast d\ast ct{r}_x\hfill \\ M_y=\frac{\sqrt{2}}{2}{C}_T\ast d\ast ct{r}_y\hfill \\ M_z=C_M\ast ct{r}_z\hfill \end{array}\right.$$

(2)

$$\left\{\begin{array}{c}ct{r}_x=-{\omega }_1^2-{\omega }_2^2+{\omega }_3^2+{\omega }_4^2=scal{e}_x\ast a_x\hfill \\ ct{r}_y=-{\omega }_1^2+{\omega }_2^2+{\omega }_3^2-{\omega }_4^2=scal{e}_y\ast a_y\hfill \\ ct{r}_z=-{\omega }_1^2+{\omega }_2^2-{\omega }_3^2+{\omega }_4^2=scal{e}_z\ast a_z\hfill \end{array}\right.$$

(3)

where $C_T$ is the lift coefficient and $C_M$ is the torque coefficient of the rotors. d denotes the length of arm of the quadrotor. $a_x,a_y,a_z\in \left[-1,1\right]$ and $ct{r}_x\in \left[-scal{e}_x,scal{e}_x\right],ct{r}_y\in \left[-scal{e}_y,scal{e}_y\right],ct{r}_z\in \left[-scal{e}_z,scal{e}_z\right]$ are intermediate variables. $scal{e}_x,scal{e}_y,scal{e}_z$ are scale constants, ${\omega }_1$ to ${\omega }_4$ are theangular velocities of the four rotors in Figure 1. $a_x,a_y,a_z$ are the results of attitude controller. The relationships between $\omega$ and $\sigma \in \left[0,1\right]$ can be modeled as a first order system:

$$\left\{\begin{array}{c}\omega =\frac{1}{T_{m}s+1}{\omega }_d\hfill \\ {\omega }_d=C_R\sigma \hfill \end{array}\right.$$

(4)

where $\omega$ and ${\omega }_d$ denote the current angular velocity and the desired angular velocity, $T_m$ is the dynamic response constant of the motor-rotor system.

The accuracy of these parameters is critical. The mass and length can be measured directly, but the measurement of other parameters is difficult. Mechanical parameters, such as moment of inertia, are directly calculated in the Solidworks software package in this paper. The physical parameters of motors and propellers are measured with a motor test beach. The remaining of unknown parameters, including $C_M$ and $T_M$, can be found in a quadrotor data base online: https://flyeval.com/[25].

A simulation environment program in which the quadrotors’ states can be updated with the control signals of motors is built with the system model and parameters. The attitude of quadrotors is represented with unit quaternions and updated with the traditional Runge-Kuta method, as shown in Equation (5).

$$\left[\begin{array}{c}\dot{q_0}\\ \dot{q_1}\\ \dot{q_2}\\ \dot{q_3}\end{array}\right]=\frac{1}{2}\left[\begin{array}{cccc}0& -{\omega }_x& -{\omega }_y& -{\omega }_z\\ {\omega }_x& 0& {\omega }_z& -{\omega }_y\\ {\omega }_y& -{\omega }_z& 0& {\omega }_x\\ {\omega }_z& {\omega }_y& -{\omega }_x& 0\end{array}\right]\left[\begin{array}{c}{q}_0\\ q_1\\ q_2\\ q_3\end{array}\right]$$

(5)

where $\left[\begin{array}{cccc}{q}_0& q_1& q_2& q_3\end{array}\right]$ is a group of unit quaternion to describe the rotation between the quadrotor body and earth frame. The quaternion in this paper is a unit quaternion:

$$\left\{\begin{array}{c}{q}_0=cos\left(\frac{\Theta }{2}\right)\hfill \\ q_1=sin\left(\frac{\Theta }{2}\right)\ast p_x\hfill \\ q_2=sin\left(\frac{\Theta }{2}\right)\ast p_y\hfill \\ q_3=sin\left(\frac{\Theta }{2}\right)\ast p_z\hfill \end{array}\right.$$

(6)

where $\Theta$ is the total rotation (the combination of roll, pitch, and yaw rotation) and $p_x,p_y,p_z$ are projections of $\Theta$ in the $x,y,z$ axes. We limit $\Theta$ between $-\pi$ and $\pi$, so $q_0$ is positive all the time. When $\Theta$ is not too large, the attitude angle of roll, pitch, and yaw can be simplified as $2\ast q_1,2\ast q_2,2\ast q_3$, respectively.

3. Quadrotor Attitude Control System Design

The attitude controller receives the desired attitude $at{t}_d$, the current attitude $att$, and the current angular velocities ${\omega }_x,{\omega }_y,{\omega }_z$ and then outputs the control values $a_x,a_y,a_z$ of each axis according to the states. The CGPs of each axis $b_{0x},b_{0y},b_{0z}$ are estimated or tuned manually. The disturbance of roll and pitch axes ${\xi }_x,{\xi }_y$ is estimated by the ESOs. Finally, the control distributor transfers $a_x,a_y,a_z$ and $t{h}_d$ to the $ct{r}_{motors}$ ($ct{r}_{motors}\in \left[0,1\right]$ and, in return, sends the attenuated control values $a_{xa},a_{ya},a_{za}$ to the ESOs because the actual control value of each axis may be attenuated by the control distributor if $t{h}_d$ is too large or too small. Figure 2 presents the control diagram.

The typical ADRC comprises three major components: the nonlinear tracking differentiator (TD), extended state observer (ESO), and state error feedback (SEF) [14]. In the traditional ADRC, the transient process is set to avoid a sharp deviation between reference signals, and the process is realized through the TD. In this paper, the transient process is automatically employed by the RL method. The CGP is the estimation of the system response capability, and the accuracy of CGP is critical to the performance of both the ESO and SEF. In previous studies, CGPs were the manually adjusted parameters. We propose a method to estimate the CGP value for each quadrotor platform with a TD. With ESOs and CGPs, the RL controller is able to control different quadrotors with the same actor network. In order to optimize the final performance, the CMA-ES algorithm is presented to tune the parameters of ESOs, and succeeded eliminating the high frequency vibration problem.

3.1. First-Order ESO and SEF for Quadrotor Attitude Control

In a quadrotor attitude control system, taking the x axis as an example, the system can be written as:

$$\left\{\begin{array}{c}\dot{\varphi }={\omega }_x\hfill \\ \dot{{\omega }_x}=b_{0x}\ast a_x+{\xi }_x\hfill \end{array}\right.$$

(7)

$$\left\{\begin{array}{c}{b}_{0x}=\frac{\sqrt{2}}{2}\ast \frac{1}{J_{xx}}\ast C_T\ast d\ast scal{e}_x\hfill \\ {\xi }_x=\frac{{\omega }_y{\omega }_z}{J_{xx}}\left(J_{yy}-J_{zz}\right)+\frac{1}{J_{xx}}{D}_x\hfill \end{array}\right.$$

(8)

where $b_{0x}$ is the estimation of the CGP and ${\xi }_x$ is the estimation of the total disturbance. The ADRC algorithm takes the nonlinearities, uncertainties, and external disturbances as the total disturbance. The total disturbance can be observed by the ESO in real time with the quadrotor state input, the control output, and an estimation of CGP. The quadrotor control system receives angular velocity data directly from the gyroscope sensors, so the form of the ESO for the quadrotor attitude control system in a single axis is a first-order ESO (Algorithm 1):

Algorithm 1 Extended state observer for quadrotor attitude control.

y: angular velocity data from sensors $z_1$: angular velocity output $z_2$: disturbance output h: time step $b_0$: critical gain parameter u: control output $e,fe$: intermediate variables $\delta ,{\beta }_{01},{\beta }_{02}$: adjustable parameters $fal$: a non-linear function [14] 1: $e=z_1-y$ 2: $fe=fal(e,0.5,\delta )$ 3: $z_1=z_1+h\ast \left(z_2-{\beta }_{01}e+b_{0}u\right)$ 4: $z_2=z_2+h\ast \left(-{\beta }_{02}\ast fe\right)$

Using the ESOs, the disturbance of three axes are estimated, and the disturbance feedback should be added to the controller output:

$$\left\{\begin{array}{c}a=a_0-\frac{{\xi }_0}{b_0}\hfill \end{array}\right.$$

(9)

where a is the final control value and $a_0$ is the output of the controller.

In this paper, only roll and pitch ESOs are designed, because the yaw axis is hardly affected by the external disturbances in our outdoor flight experience. In severe flight situations, such as propeller breakage and load swinging, the control system should focus on maintaining the balance of the roll and pitch axes, and the performance of the yaw axis may not be so important.

3.2. Estimation of the CGPs

In this paper, the CGP is the proportional relation between the control signal $a_x,a_y,a_z$ and the angular accelerations in three axes $\dot{{\omega }_x},\dot{{\omega }_y},\dot{{\omega }_z}$. The accuracy of the CGPs is critical to both the RL controller and ESO feedback. The CGPs of the F550 quadrotor can be simply calculated with Equation (8), but the measurement of different quadrotors or platforms can be difficult and expensive and, thus, a method to estimate the CPGs is presented in this section.

As Equation (7) shows, the CGP can be calculated given the state of a quadrotor, including angular acceleration, control value, and disturbance. When considering the ill-conditioned equation problem and noise of sensors, the system should obtain enough excitation during the estimation process, but too much excitation may cause a quadrotor crash. To precisely estimate angular acceleration data, a TD is designed to calculate the differential of angular velocity data in Algorithm 2:

Algorithm 2 Tracking differentiators for angular velocity data.

$v_0$: angular velocity data from sensors $v_1$: angular velocity output $v_2$: angular acceleration output $fhan$: a non-linear function [14] $e,fh$: intermediate variables h: time step $h_0$: adjustable parameters, usually two or three times h 1: $e=v_1-v_0$ 2: $fh=fhan(e,v_2,r_0,h_0)$ 3: $v_1=v_1+h\ast v_2$ 4: $v_2=v_2+h\ast fh$

The estimation process starts when the quadrotor hovers stably above an open and wide ground. The controller produces an excitation $a_e\in \left[0.3,0.5\right]$ for $t_e=0.1$s on a single axis, and then the CGP in this axis can be calculated with these flight data.

3.3. Single-Parameter-Tuned Attitude Control Method

The RL method is a frame with these key factors: the agent, environment, state ${\mathit{s}}_t$, policy $\pi \left({\mathit{s}}_t\right)$, action ${\mathit{a}}_t$, reward $r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)$, and state-action value $V_t\left({\mathit{s}}_t,{\mathit{a}}_t\right)$. During the training process, the agent takes action ${\mathit{a}}_t$ according to state ${\mathit{s}}_t$ and policy $\pi$ in the environment, and obtains a reward $r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)$ in return. Value $V_t$ is the sum of the expected future rewards:

$$V_t=r_t+\lambda r_{t+1}+{\lambda }^2{r}_{t+1}+\dots =r_t+\lambda V_{t+1}$$

(10)

where discount factor $\lambda \in \left[0,1\right)$ is introduced to prevent the infinite sum of future rewards. However, the future reward is unknown, so we assume that an optimal policy ${\pi }_{opt}$ can maximize $V_{t+1}$:

$$\begin{array}{cc}\hfill V_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)& =r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)+\lambda ma{x}_{{\mathit{a}}_t+1\in A}\left(V_{t+1}\left({\mathit{s}}_{t+1},{\mathit{a}}_{t+1}\right)\right)\hfill \\ \hfill & =r_t\left({\mathit{s}}_{t+1},{\mathit{a}}_t\right)+\lambda V_{t+1}\left({\mathit{s}}_{t+1},{\pi }_{opt}\left({\mathit{s}}_{t+1}\right)\right)\hfill \end{array}$$

(11)

where A is the action space. During the training process, we use the current policy $\pi$ instead, and this process can make the performance of $\pi$ gradually close to ${\pi }_{opt}$.

With the ADRC methods, the quadrotor attitude control is divided into three second-order linear systems with different CGPs and disturbances, so we lock the pitch and yaw axis of a quadrotor as a standard second order linear rotation system rather than a full quadrotor. The agent is similar to an inverted pendulum, which explores the rotation space while discovering the optimal policy and tracking the desired roll angle. Accordingly, the system state variables in training process are represented as: $\theta$ is the attitude angle, $\omega$ is the angular velocity, $b_0$ is the CGP of this axis and $a_0$ is the control output. Actuator delay is also considered as a first order system in Equation (4). During the training process, no disturbance is added in the simulation.

3.3.1. Network Structures

The RL controller in this paper is based on the actor-critic scheme [26], in which the critic network judges the value of a state-action-pair, and the actor network outputs the control values according to the states input. The state in the attitude control process consists of the error between the desired attitude and the current attitude, along with the current angular velocity. The CGP is added to the state in this paper, so the state list is written as: $\mathit{s}=\left[\theta ,\omega ,b_0\right]$. The action list is expressed as $\mathit{a}=\left[a\right]$, where $a\in \left[-1,1\right]$ is the control value in the controlled axis.

We have two critic networks with the same structure: target critic network and online critic network [26]. The critic networks are fully connected networks (FCNs), with a hidden layer of 32 nodes and ReLU activation function. All of the experience is used to train the online critic network to fully observe the system and compute the value $\left[V\right]$. After that, we define the cost function as a squared error function and lower the cost using the Adam optimizer. The target critic network is soft updated with an online critic network.

The online actor network and target actor network are also FCNs with a 32-node ReLU hidden layer. The tanh activation function is set in the output layer to limit the output between −1 and 1. The state lists are sent directly to the actor network and action lists are computed. The gradients of the online actor network are computed with sampled policy gradients directly and then update the weights with the Adam optimizer [27]. The target actor network is soft updated [26] with the online actor network as well. Figure 3 presents the structures of actor networks and critic networks.

3.3.2. Reward Function Shaping and Mirror Sampling Methods

Reward function shaping and mirror sampling methods are presented in order to eliminate the steady-state error problem of RL algorithms in this section.

Traditionally, the reward functions are squared error functions such as $r\left(\mathit{s}\right)=-a_1\ast {\theta }^2-a_2\ast {\omega }^2$, where $a_1$ and $a_2$ are positive parameters. However, these squared error reward functions lead to slow convergence rates and severe steady-state error problems in our former experiments. From the perspective of this paper, the state with a low angle error or low angular velocity should be rewarded positively rather than be punished with less penalty. This paper presents a simple but powerful piecewise function $fr\left(x,\delta ,\rho \right)$ to meet this demand:

$$fr\left(x,\delta ,\rho \right)=\left\{\begin{array}{cc}-\frac{x}{\delta }+1,\hfill & \left(\left|x\right|\le \delta \right)\hfill \\ \frac{x-\delta }{\delta -\rho },\hfill & \left(\delta <\left|x\right|<\rho \right)\hfill \end{array}\right.$$

(12)

where x is the angle error or angular velocity to be rewarded, $\rho$ is the upper level of x, and $\delta$ is the boundary line. If $\left|x\right|\le \delta$, then x should be rewarded positively. With $fr\left(x,\delta ,\rho \right)$, the reward function is designed as:

$$r\left(s\right)=-a_1\ast fr\left(\theta ,{\delta }_1,{\rho }_1\right)-a_2\ast fr\left(\omega ,{\delta }_2,{\rho }_2\right)$$

(13)

where $a_1,a_2,{\sigma }_1,{\rho }_1,{\sigma }_2$ and ${\rho }_2$ are positive parameters to be tuned.

Because the quadrotor attitude control system is a symmetrical system, whenever a list of experience $E=\left(\mathit{s},\mathit{a},{\mathit{s}}^{\prime }\right)=\left(\left[\theta ,\omega ,b_0\right],\left[a\right],\left[{\theta }^{\prime },{\omega }^{\prime },b_0\right]\right)$ is explored, the mirror sample experience $E_m=\left({\mathit{s}}_m,{\mathit{a}}_m,{\mathit{s}}_m^{\prime }\right)=\left[-\theta ,-\omega ,b_0\right],\left[-a\right],\left[-{\theta }^{\prime },-{\omega }^{\prime },b_0\right]$ is also correct. Moreover, the zero state-action point $E_o=\left(s_o,a_o,s_o^{\prime }\right)=\left[0,0,b_0\right],\left[0\right],\left[0,0,b_0\right]$ is an obvious result, but may never be explored in the training process, which is an important reason for the steady-state error problem. These samples should be considered to accelerate the training process and eliminate steady-state errors.

3.3.3. Training Algorithm

The training algorithm in this paper is the DPG algorithm [26], and we add some Gaussian noise to the action as the exploration strategy. Algorithm 3 lists the whole algorithm.

Algorithm 3 Training algorithm with mirror sampling.

$e_{max}$: max number of episodes $s_{max}$: max number of steps in an episode $n_s$: max number of samples $R\left({\mathit{s}}^{\prime }\right)$: reward function in Equation (13) $\tau$: soft update rate 1: initialize weights of online critic network ${\mathbf{\theta }}_v$ and online actor network ${\mathbf{\theta }}_{\pi }$ randomly 2: initialize weights of target critic network ${\mathbf{\theta }}_v^{\prime }$ and target actor network ${\mathbf{\theta }}_{\pi }^{\prime }:{\mathbf{\theta }}_v^{\prime }\leftarrow {\mathbf{\theta }}_v,{\mathbf{\theta }}_{\pi }^{\prime }\leftarrow {\mathbf{\theta }}_{\pi }$ 3: initialize memory box B 4: for$epi=1,2\dots e_{max}$do 5:     initialize the agent with random state 6:     put the zero state-action point $E_o$ into the memory box $B\leftarrow B\bigcup E_o$ 7:     for $s=1,2\dots s_{max}$ do 8:         take action $\mathit{a}$ according to the exploration strategy and current state $\mathit{s}$ 9:         receive the next state ${\mathit{s}}^{\prime }$ from the environment 10:       put E and the mirror sample $E_m$ into the memory box $B\leftarrow B\bigcup E\bigcup E_m$ 11:       $T=T\cup T_i;$ 12:       for $i=1,2\dots n_s$ do 13:            sample experience ${\mathit{e}}_i=\left({\mathit{s}}_i,{\mathit{a}}_i,{\mathit{s}}_i^{\prime }\right)$ from B 14:            compute reward of experience ${\mathit{e}}_i:r_i\leftarrow R\left({\mathit{s}}_i^{\prime }\right)$ 15:            $y_i\leftarrow r_i+\lambda V\left(\left({\mathit{s}}_i^{\prime },\pi \left({\mathit{s}}_i^{\prime }|{\mathbf{\theta }}_{\pi }\right)\right)|{\mathbf{\theta }}_v\right)$ 16:            update ${\mathbf{\theta }}_v$ by minimizing the loss: $L=\frac{1}{N}{\sum }_i{\left(y_i-V\left({\mathit{s}}_i,{\mathit{a}}_i|{\mathbf{\theta }}_v\right)\right)}^2$ 17:            update ${\mathbf{\theta }}_{\pi }$ using the sampled policy gradients: ${▽}_{{\mathbf{\theta }}_{\pi }}\approx \frac{1}{N}{\sum }_i{▽}_{\mathit{a}}V\left(\left(\mathit{s},\mathit{a}|{\mathbf{\theta }}_v\right){|}_{\mathit{s}={\mathit{s}}_i,\mathit{a}={\mathit{a}}_i}{▽}_{{\mathbf{\theta }}_{\pi }}\pi \left(s|{\mathbf{\theta }}_{\pi }\right)\right){|}_{\mathit{s}={\mathit{s}}_i}$ 18:            soft update ${\theta }_v^{\prime }$ with ${\mathbf{\theta }}_v:{\mathbf{\theta }}_v^{\prime }\leftarrow \tau {\mathbf{\theta }}_v+\left(1-\tau \right){\mathbf{\theta }}_v^{\prime }$ 19:            soft update ${\theta }_{\pi }^{\prime }$ with ${\mathbf{\theta }}_{\pi }:{\mathbf{\theta }}_{\pi }^{\prime }\leftarrow \tau {\mathbf{\theta }}_{\pi }+\left(1-\tau \right){\mathbf{\theta }}_{\pi }^{\prime }$ 20:         end for 21:     end for 22: end for

3.4. ESO Parameter Tuning with CMA-ES Algorithm

Although there are some rules about parameter tuning in ESO, it is still repetitive work, even for designers with considerable experience. The parameters to tune are a list of $\left[{\beta }_{01},{\beta }_{02}\right]$ in Algorithm 1 for the roll and pitch axes. In this section, only the ESO in the roll axis is optimized.

In the simulation environment, the disturbance can be directly obtained, so we optimize the process during flight, and the controller is the actor network trained in Section 3.3, along with the disturbance feedback that is calculated by the ESO. A task is designed to test and quantify the performance of the ESO. The system starts on the zero point state, and the controller tries to hold the system on the zero point state for a period of time, with the same time-varying square wave disturbance added to the system in each test. The costs in each task are computed with the error of ESO output ${\xi }_0$ and disturbance $\xi$ in every time step during the task.

In the real quadrotor platform, we designed a specific test: the quadrotor is placed on the ground with no external disturbance, and control signal $a_x$ is sent to the controller; but, the motors are not actuated and remain still during the test. It is worth noting that the control signal $a_x$ here is not computed by the controller in Section 3.3; we set $a_x$ as a square wave. In this situation, from the perspective of the controller, a disturbance compensates for the control signals to keep the quadrotor still, so the value of disturbance $\xi$ is just the negative of current control signal $a_x$ multiplied by the CGP $b_0$. Using this test, we avoid problems, such as safety risks, computational costs, and battery capacity limitation, etc. The task for real quadrotor optimization is that the system starts on the zero point state and holds still for a period of time, and the same time-varying control signal $a_x$ is added to the system in each test. The costs in each task are also computed with the error of ${\xi }_0$ and $\xi =-a_x\ast b_0$ in every time step during the task.

We present ESO parameter tuning with the CMA-ES algorithm in this section. The CMA-ES algorithm is a stochastic or randomized method for real-parameter (continuous domain) optimization of nonlinear, nonconvex functions. The CMA-ES algorithm consists of five parts: sampling, calculating costs, updating the mean value, adapting the covariance matrix, and step-size control [24].

3.4.1. Sampling

In the CMA-ES, a population of $\lambda$ new search points (individuals, offspring), is generated by sampling a multivariate normal distribution $N\left({\mathit{m}}_g,{\sigma }_g^2{\mathit{C}}_g\right)$:

$$\left\{\begin{array}{c}{\mathit{x}}_i={\mathit{m}}_g+{\sigma }_g{\mathit{y}}_i,{\mathit{y}}_i\sim N\left(0,{\mathit{C}}_g\right)\hfill \\ {\mathit{y}}_i=\mathit{B}\mathit{D}{\mathit{z}}_i,{\mathit{z}}_i\sim N\left(0,\mathit{I}\right)\hfill \end{array}\right.fori=1,2\dots \lambda$$

(14)

The parameter $m_g$ is the mean value, ${\sigma }_g$ is the step-size (scale value), ${\mathit{x}}_i$ is a search point sampled from multivariate normal distribution $N\left({\mathit{m}}_g,{\sigma }_g^2{\mathit{C}}_g\right)$, and ${\mathit{y}}_i$ is sampled from $N\left(0,{\sigma }_g^2{\mathit{C}}_g\right)$. $\mathit{B},\mathit{D}$ results from an eigendecomposition of the covariance matrix ${\mathit{C}}_g=\mathit{B}\mathit{D}\mathit{D}{\mathit{B}}^T$. The columns of $\mathit{B}$ are an orthonormal basis of the eigenvectors. The diagonal elements of the diagonal matrix $\mathit{D}$ are the square roots of the corresponding positive eigenvalues. In fact, CMA-ES controls the search directions with the matrix $\mathit{D}$, and controls the search range with $\mathit{B}$ and ${\sigma }_g$.

3.4.2. Calculating Costs

It is critical to lower the error between the real disturbance $\xi$ and the estimation output of the ESO ${\xi }_0$ in order to eliminate the high frequency vibration problem. The tasks in the simulation and real quadrotor platform presented above are tested for all search points. For every search point ${\mathit{x}}_i$, the cost in every time step is computed with $cos{t}_{step\left(\xi ,{\xi }_0\right)}=\sqrt{\left|\xi -{\xi }_0\right|}$, and the total cost $cost\left({\mathit{x}}_i\right)$ is the sum of all costs in every time step.

3.4.3. Updating the Mean Value

The mean value of next generation ${\mathit{m}}_{g+1}$ is updated based on the costs above. We sort these costs and only the first $\mu \left(\mu <\lambda \right)$ samples are recorded: $cost\left({\mathit{x}}_{1:\mu }\right)\le cost\left({\mathit{x}}_{2:\mu }\right)\le \dots \le cost\left({\mathit{x}}_{\mu :\mu }\right)$. A list of weights ${\omega }_1>{\omega }_2>\dots >{\omega }_{\mu },\left({\sum }_{i=1}^{\mu }{\omega }_i=1\right)$ is designed in the beginning of the algorithm, and ${\mathit{m}}_{g+1}$ is calculated with these weights:

$${\mathit{m}}_{g+1}={\mathit{m}}_g+\sum _{i=1}^{\mu }{\mathit{x}}_{i:\mu }\ast {\omega }_i$$

(15)

3.4.4. Adapting the Covariance Matrix

The covariance matrix ${\mathit{C}}_g$ determines the search directions. Two methods to update the ${\mathit{C}}_g$: rank-1 update and rank-$\mu$ update are presented in this section. The rank-1 update means that we should search new points in the direction of the new mean value:

$${\mathit{p}}_{g+1}=\left(1-c\right){\mathit{p}}_g+\sqrt{c\left(2-c\right)}\sqrt{{\mu }_{\omega }}\frac{{\mathit{m}}_{g+1}-{\mathit{m}}_g}{{\sigma }_g}$$

(16)

where ${\mathit{p}}_g$ is the evolution path, c is the learning rate of the evolution path update, and ${\mu }_{\omega }=\frac{1}{{\sum }_{i=1}^{\mu }{\omega }_i^2}$. The rank-$\mu$ update means we should search new points in the direction of the search points with low costs. As a result, the covariance matrix ${\mathit{C}}_g$ updates with the equation below:

$${\mathit{C}}_{g+1}=\left(1-c_1-c_{\mu }\right){\mathit{C}}_g+c_1{\mathit{p}}_{g+1}{\mathit{p}}_{g+1}^T+c_{\mu }\sum _{i=1}^{\mu }\left(\frac{{\mathit{x}}_{i:\mu }-m_g}{{\sigma }_g}\right){\left(\frac{{\mathit{x}}_{i:\mu }-m_g}{{\sigma }_g}\right)}^T$$

(17)

where $c_1$ and $c_{\mu }$ are the learning rates for the rank-1 update and the rank-$\mu$ update.

3.4.5. Step-Size Control

This section presents a simple method to control the step size ${\sigma }_g$. We scale ${\sigma }_g$ according to the mean value $cost\left({\mathit{m}}_g\right)$:

$${\sigma }_g=\left\{\begin{array}{cc}{\sigma }_{max}\ast \frac{cost\left({\mathit{m}}_g\right)}{cos{t}_{max}},\hfill & ifcost\left({\mathit{m}}_g\right)<cos{t}_{max}\hfill \\ {\sigma }_{max},\hfill & ifcost\left({\mathit{m}}_g\right)\ge cos{t}_{max}\hfill \end{array}\right.$$

(18)

where ${\sigma }_{max}$ is the max step-size, and $cos{t}_{max}$ is the max cost we set.

Algorithm 4 lists the total algorithm:

Algorithm 4 Covariance matrix adaptation evolution strategy (CMA-ES) algorithm for extended state observer (ESO) parameter tuning.

$g_{max}$: max number of generations 1: for$g=1,2\dots g_{max}$do 2:     compute $\mathit{B}$,$\mathit{D}$ from ${\mathit{C}}_g$ 3:     for $i=1,2\dots \lambda$ do 4:         sample point ${\mathit{x}}_i$ with $\mathit{B}$,$\mathit{D}$ and $\mathit{I}$ 5:         compute the cost of sample point $cost\left({\mathit{x}}_i\right)$ 6:     end for 7:     sort the points based on the costs: $cost\left({\mathit{x}}_{1:\mu }\right)\le cost\left({\mathit{x}}_{2:\mu }\right)\le \dots \le cost\left({\mathit{x}}_{\mu :\mu }\right)$ 8:     update the mean value using Equation (15) 9:     update the evolution path using Equation (16) 10:   update the covariance matrix using Equation (17) 11:   update the step-size using Equation (18) 12: end for

4. Training and Testing in the Simulation

4.1. Training Environment

A simulation environment is developed to train and test the UAV controller. The program is coded with Python 3.8 and TensorFlow 1.14. Quadrotors with different physical parameters can be created, and their performance can be tested in the environment. The training and testing processes are conducted on a laptop computer with an NVIDIA GTX1060 graphics processing unit (GPU) and i7-7700HQ central processing unit (CPU).

4.2. Performance in the Simulation and Details

4.2.1. Quadrotor Details

The quadrotor UAV used in this paper is an “X” type F550 quadrotor, equipped with TMOTOR 2213 brushless motors, TMOTOR 9545 propellers, SKYWALKER 30A Electronic Speed Controls (ESCs) and an ACE 5300mAh 30C 3S Li-polymer battery. The adjustable timing of the ESC is set to a medium level to achieve better performance. With these parameters, the CGPs of the three axes can be calculated.

Table 1. Physical parameters of the quadrotor.

Parameter	Value
Mass m	$1.311$ kg
Length of arm d	$0.24$ m
Moment of inertia $J_{xx}$	$1.762\ast {10}^{-2}$ kg·m${}^2$
Moment of inertia $J_{yy}$	$1.796\ast {10}^{-2}$ kg·m${}^2$
Moment of inertia $J_{zz}$	$2.805\ast {10}^{-2}$ kg·m${}^2$
Lift coefficient $C_T$	$9.138\ast {10}^{-6}$ N/(rad/s)${}^2$
Torque coefficient $C_M$	$1.368\ast {10}^{-7}$ N·m/(rad/s)${}^2$
Motor dynamic response constant $T_m$	$0.0157$ s
Motor linear control constant $C_R$	$973.6$ rad/s
Control scale in x and y axes $scal{e}_x$,$scal{e}_y$	$6.5\ast {10}^5$
Control scale in z axis $scal{e}_z$	$7.3\ast {10}^5$
CGP of x and y axes	$66.0$
CGP of z axis	$3.6$

lists the system parameters.

We estimated other types of quadrotors such as an F250 traversing machine and a 10 kg grade agricultural plant protection quadrotor. The CGPs of the x and y axes are between 50 and 200, and the CGPs of the z axes are between 3 and 10. The range of CGPs is limited between 1 and 200 in this paper as a result.

4.2.2. Performance of the RL Controller and Training Details

Taking the roll and pitch attitude control $\left(cgp\approx 66.0\right)$ as an example, the step response and sine function tracking performance is shown in Figure 4:

The performance is excellent with a high convergence rate and nearly no overshoot and steady-state error. In theory, the RL controller grows close to the global optimal controller with sufficient proper training, which performs better than any other traditional controller.

The controller in this paper has great robustness for different types of quadrotors with different CGPs. We test the performance of quadrotors with different CGPs (cgp = 6.0, 66.0, 100.0, 180.0) in Figure 5. The quadrotors start with random states (attitude angles and angular velocities) and recover to zero points with the RL controller, and this process is tested for 100 times. The network here is fully trained for 1000 episodes. All of the quadrotors perform excellently and we can conclude that the rate of convergence improves with larger CGP value.

The training process improves the performance gradually, and the performance of controllers in different episodes (ep = 0, 100, 200, 1000, respectively) is tested in Figure 6. The same quadrotor (cgp = 66.0) is tested here to recover from random initial states 100 times. The controller functions better during the training process as a result.

The reward shaping and mirror sampling method greatly accelerates the training process and improves the control quality. In Figure 7, the reward function of this RL controller is a traditional squared error function $r\left(\mathit{s}\right)=-{\theta }^2-0.1{\omega }^2$, and the controller is fully trained with 1000 episodes. The performance of this RL controller without the reward shaping and mirror sampling method is not so good in the $\pm {30}^{\circ }$ step response task and sine function tracking task as compared to the RL controller in Figure 4; the response rates are too low and the steady-state errors are obvious.

The total training process took approximately 12 min. The hyperparameters of the training process are listed in

Table 2. Reinforcement learning (RL) training hyper parameters.

Parameter	Value
Discount factor $\gamma$	$0.9$
Critic net learning rate ${\alpha }_{critic}$	$0.001$
Actor net learning rate ${\alpha }_{actor}$	$0.001$
Batch size N	64
Soft update rate $\tau$	$0.01$
Standard deviation of Gaussian noise	$0.3$
Maximum number of training episodes $e_{max}$	1000
Control time step $dt$	$0.01$ s
Maximum time step in an episode $t_{max}$	5 s
Reward function parameter $a_1$	$1.0$
Reward function parameter ${\delta }_1$	$0.01$
Reward function parameter ${\rho }_1$	$\pi$
Reward function parameter $a_2$	$0.1$
Reward function parameter ${\delta }_2$	$0.1$
Reward function parameter ${\rho }_2$	$2\pi$

.

4.2.3. CMA-ES Training Detail in the Simulation Environment

In this section, square wave disturbances are added to the system. The system with no ESO feedback cannot overcome the effect of these disturbances, as the first picture in Figure 8 shows. CMA-ES searches for better $\left[{\beta }_{01},{\beta }_{02}\right]$ to reduce the cost, and the performance of the total controller also improves. In Figure 8, the CMA-ES algorithm improves the performance gradually as the generation grows ($gen=0,1,10,30,100$ respectively) and finally converges to $\left[{\beta }_{01}=110.0,{\beta }_{02}=1460.0\right]$.

The total training process took approximately 16 min. The hyperparameters are listed in

Table 3. CMA-ES training hyper parameters in simulation.

Parameter	Value
Generation number $g_{max}$	100
Initial standard deviation ${\sigma }_{max}$	60
Number of sample points $\lambda$	100
Number of offspring points $\mu$	25
Learning rate of evolution path update c	$0.1$
Learning rate of rank 1 update $c_1$	$0.2$
Learning rate of rank $\mu$ update $c_{\mu }$	$0.5$
Time step	$0.01$ s

.

4.2.4. Performance of the Full Controller

The quadrotor attitude controller combined RL with ADRC in this paper has excellent performance, including a fast convergence rate, nearly no overshoot and steady-state error, a high robustness for different platforms, and a strong disturbance rejection ability. In Figure 9, the quadrotors with different CGPs perform well in the step response test with the same external disturbance. The disturbance is a square wave, but the final performance compensates for the disturbance completely in a timely manner. The overshoot angle and final steady-state errors are less than 0.001 degrees, which can be ignored. The 30 degrees step-response take less than 0.35 s, which show great convergence rate. In conclusion, the performance in Figure 9 meet our demand for a quadrotor attitude control system. The performance of these quadrotors also indicate that the convergence rate and the disturbance rejection ability of a quadrotor can be improved with a larger CGP value.

We designed a traditional PID controller for comparison. The PID parameters are also optimized with the CMA-ES algorithm, and this controller can be considered as a near-optimal PID controller. In Figure 10, the PID controller is not as good as that of our RL controller. The control quality of the PID controllers is limited due to the conflict between rapidity and stability, and the RL controller can solve the problem with enough exploration during training. When compared to the traditional PID controller, the RL controller in this paper only has one parameter to tune, which is a great advantage.

Notably, parameter I in the PID controller converges to 0 during the CMA-ES optimization process. However, the integral controller can also compensate for disturbance over time. We manually set parameter I as a small value in the next experiment. As Figure 11 shows, if the external disturbances are added to the system, then the PID controller is unable to compensate for these errors in a timely manner, and the compensation efficiency of integral controllers is not sufficient.

In conclusion, the controller in this paper has shown greater advantages than the traditional PID controller in control performance, robustness for system models, and external disturbance.

5. Experiments

The flight controller is based on a Raspberry Pi 3B microcomputer whose CPU has four 1.2 GHz cores, which is capable of running our actor network inference in real time. The program is coded with C++ and it runs on the Raspbian operating system, which is not a real-time system, but we did substantial work to reduce the latency of the system. For example, there are four cores on the Raspberry Pi board, and we stop running the Raspbian operating system on the fourth core, which is used wholly to run the flight control program. The power consumption of this Raspberry Pi is less than 10w generally, which can be ignored as compared to the motors (about 80w for each motor during hovering state). The actor network is coded with the Eigen library and it took less than 6 $\mathsf{\mu }s$ to compute the network for a given state. Using these skills and techniques, we can run the CMA-ES algorithm to tune the ESO parameters on the real quadrotor directly and conduct outdoor flight experiments.

5.1. CMA-ES Training Result and Detail on Real Platform

The ESO parameters tuning with the CMA-ES experiment is also conducted on the real quadrotor platform. Square wave disturbances are added to the system. CMA-ES searches for better $\left[{\beta }_{01},{\beta }_{02}\right]$ to reduce the error between the ESO output and the real disturbance. In Figure 12, the total costs decrease as the generation increases, and the mean value finally converges to $\left[{\beta }_{01}=112.0,{\beta }_{02}=1080.0\right]$.

The optimization process took approximately 40 min. on a Raspberry Pi 3B microcomputer. The hyperparameters are listed in

Table 4. CMA-ES training hyper parameters on real platform.

Parameter	Value
Generation number $g_{m}ax$	36
Initial standard deviation ${\sigma }_{m}ax$	60
Number of sample points $\lambda$	10
Number of offspring points $\mu$	3
Learning rate of evolution path update c	$0.1$
Learning rate of rank 1 update $c_1$	$0.2$
Learning rate of rank $\mu$ update $c_{\mu }$	$0.5$
Time step	$0.01$ s
Total time of a task	5 s

. When compared to the training process in the simulation environment, the number of sample points and offspring points are greatly reduced to accelerate the training process.

5.2. Outdoor Flight Experiment Result

Two flight tests are shown in this section. The quadrotor platform, the controller, and the flight performance are shown in Figure 13.

(1)

Attitude Tracking Performance.

In this test, a set of experiments are designed in order to evaluate the tracking performance of the proposed method. No external disturbance is added in this test. The desired attitudes are manually controlled with a remote controller, and the flight log data are presented in Figure 14 and Figure 15. According to Figure 14, the actual attitude can track the desired attitude rapidly and smoothly. Figure 15 shows the attitude error are between four degrees during the test.

(2)

Disturbance Rejection Performance.

A bottle of water is hung on one side of the drone in this test (see Figure 13). Notably, the water bottle keeps swinging during the flight experiment, which can be considered to be a severe unknown disturbance. Figure 16 shows the tracking performance is still good with the bottle swinging. Figure 17 shows that the maximum attitude error is below four degrees even with severe disturbance.

6. Conclusions

In this paper, we present a single-parameter-tuned attitude control for a quadrotor in the presence of an unknown disturbance, which is, only one tuning parameter is required for each axis. The method is a combination of RL and ADRC. The critical parameters of the system are extracted as CGPs and used to extend the RL state list to improve the robustness of the RL controller. The introduction of mirror sampling and reward shaping eliminate steady-state errors that are incurred using the RL algorithm during continuous control and considerably accelerate the training process. ESOs are designed to estimate external disturbances. A CMA-ES algorithm is developed to automatically tune the ESO parameters, which successfully optimizes the controller performance for both a simulation environment and a real quadrotor platform. The complete controller exhibits excellent performance and remains robust for quadrotors with different physical parameters. When compared to model-based algorithms, such as BS, sliding mode, and H-infinity control algorithms, the method in this paper can be applied on quadrotors with unknown models and parameters meanwhile remain good performance. When compared to PID-based algorithms, our method greatly simplify the parameter tuning work and show better performance.

In our future work, more quadrotor platforms will be tested in order to verify the effectiveness of this single-parameter-tuned quadrotor attitude control system. We will also focus on the improvement of rapidity and stability of the quadrotor attitude control system, and the network training details will be modified further. Moreover, the methods will be tested in quadrotor position control systems.