Approximate Optimal Tracking Control for Partially Unknown Nonlinear Systems via an Adaptive Fixed-Time Observer

Abstract

This paper investigates a novel adaptive fixed-time disturbance observer (AFXDO)-based approximate optimal tracking control architecture for nonlinear systems with partially unknown dynamic drift and perturbation under an adaptive dynamic programming (ADP) scheme. To attenuate the impact of disturbance, a novel AFXDO was designed based on the principle of a fixed-time stable system without prior information of disturbance, making disturbance observer errors converge to zero in a fixed time independent of initial estimation error. Additionally, approximate optimal control is conducted by incorporating the real-time estimation of AFXDO into a critic-only ADP framework to stabilize the dynamics of tracking errors and strike a balance between consumption and performance. In particular, to address the heavy calculation burden and oscillation phenomenon in the traditional actor–critic structure, an improved adaptive update law with a variable learning rate was developed to update the weight for adjusting the optimal cost function and optimal control policy simultaneously, avoiding the initial chattering phenomenon and achieving a prescribed convergence without resorting to dual networks. With the efforts of AFXDO and a weight law with a variable learning rate, the track errors were achieved with fast transient performance and low control consumptions in a fixed time. By revisiting Lyapunov stability, the tracking error and weight estimation error were proven to be uniformly ultimately bounded, and the designed control tended to optimal control. The simulations were carried out on quadrotor tracking to demonstrate the effectiveness of the developed control scheme, which achieves rapid convergence by lower control consumption in 4 s, where the cost function is reduced by 19.13%.

1. Introduction

In engineering applications, tracking control for nonlinear systems has been treated as an ever-increasing hot research topic over the last few years by virtue of its extensive application scenarios including the robot [1], permanent magnet synchronous motor [2], unmanned surface vehicle [3], and quadrotor [4]. Due to the inevitable presence of unknown system dynamics or uncertain parameters with the property of symmetry or asymmetry, as well as the external disturbances induced by working scenarios, the investigated model with uncertain system dynamic drift and extraneous disturbances widely exists. Conventional proportional integral control can hardly suppress the disturbance and dynamic drift to guarantee a robust tracking performance. To date, existing control methods to enhance the tracking performance for nonlinear systems can be roughly classified as sliding mode control [5], adaptive control [6,7], etc. The neural network and fuzzy logic system are two tools to make decisions for partially unknown systems, and their effectiveness can be verified in many areas, such as [6,8,9,10]. There have been many studies devoted to improving the estimation performance of neural networks and fuzzy logic systems [11,12,13,14,15,16,17,18,19]. However, the heavy computation burden caused by FLS and NN with adjusting parameters is impractical in real-world applications.

Aside from the classical robust control to handle partially unknown nonlinear systems, many scholars have shed light on disturbance observer (DO) for nonlinear uncertain systems on account of its concise form and validity. Its principal advantage is that DO-based controllers not only significantly improve the anti-interference ability but also retain the desired control performance among theories and practical engineering applications [20,21,22,23]. For example, approximate tracking control is put forward with neural network (NN) and nonlinear DO to approximate the system uncertainties [20]. In [22], a sliding mode control was designed for spacecraft to regulate attitude, where a novel integral DO with an extra integral term was inserted to estimate the lumped disturbances with high precision. In [23], an improvement of the reduced-order proportional integral observer was considered for feedforward compensation to avoid the effect caused by disturbance. Although there emerge various DOs focusing on disturbance attenuation, the convergence results of observation errors can only be assured when the time tends to infinity, which is uneconomic for real-world plants in engineering applications.

Therefore, a novel disturbance observer is developed, while an optimal tracking controller is proposed with a variable learning rate weight law. The investigated framework can achieve high precision and the rapid convergence of tracking performance with lower control consumption by designing the following goals:

• An adaptive fixed-time disturbance observer (AFXDO) is put forward for the estimation of lumped disturbance;
• The controller is set up based on the proposed AFXDO;
• The weight law with a variable learning rate is proposed to avoid the oscillation phenomenon during the learning process.

2. Related Works

Thus, aiming to achieve zero-disturbance within an effective short time and obtain fast tracking performance, finite-time stability with adjustment settling time has received considerable attention [24,25,26,27,28], which means the time from the initial state to the stable state is adjustable. In [24], a new super-twisting algorithm is provided for high-order systems with matched and mismatched disturbances, where a finite-time DO was designed to enable disturbance estimation errors to converge to zero in a finite time. For a second-order multiagent system, a centralized controller was developed in [26] based on the estimation of disturbance, where a finite-time observer was proposed for each agent to estimate disturbance. In [28], adaptive control was developed for rigid spacecraft to achieve attitude-tracking performance, where actuator faults are estimated by designing finite-time DO in the spirit of an improved finite-time stable system. Note that finite-time DOs are preferable to ensure higher accuracy and stronger robustness than conventional DO with asymptotic convergence. However, the settling time of finite-time DO is closely related to initial estimation error, which cannot be estimated without the known initial circumstance. Moreover, the settling time will tend to infinity when the initial estimation is far from the actual perturbation, which promotes further extension of finite-time stability, named the fixed-time technique, explored by Polykov [29]. In terms of disturbance attenuation, fixed-time DO is featured with fixed-time stability, i.e., the settling time of estimation error is dominated chiefly by control parameters rather than the initial estimation error [30,31,32]. By importing a fixed-time disturbance observer, a novel composite control consisting of a fixed-time controller and an anti-saturation controller was developed in [30] for an uncertainties model with actuator saturation to guarantee trajectory tracking and attitude regulation. To enhance the robustness of disturbance, tracking control for velocity and attitude was conducted by non-smooth backstepping control and fixed-time DO to compensate for the lumped disturbances [31]. In [32], a fixed-time DO was developed to approximate lumped perturbations by utilizing the proposed new fast fixed-time stable system. Note that the published outcomes related to fixed-time stability are subject to the following defects: One is that the observer gains are always selected and associated with the upper bound of disturbance and its derivative to assure stability. Another is that it is inferior in handling performance optimization. Thus, in the absence of an upper bound of disturbance and its derivative, how to develop a novel control policy capable of performance optimization and fixed-time disturbance attenuation deserves deep investigation.

Reinforcement learning (RL) provides an efficient way to derive performance optimization by pursuing the maximum cost function, which offers a numerical method for solving optimal control. In the control field, it is equivalent to approximate dynamic programming (ADP). Its core is to solve the Hamilton–Jacobi–Bellman (HJB) equation for approximating optimal policy and corresponding optimal cost function via NN approximators, avoiding the curse of dimensionality existing in solving the optimal control of nonlinear systems. In particular, the optimal cost function and optimal control policy are estimated with an actor–critic dual networks framework. For example, the nonlinear control is designed for a nonlinear perturbation system, where the worst-case disturbance is attenuated by incorporating zero-sum differential game theory into the ADP scheme [33]. Note that the worst case is considered in control with over-conservative properties. To achieve disturbance attenuation, there emerges a wide range of results related to the ADP technique. For instance, the identifier–critic–actor architecture is developed in [34] with a fuzzy logic system to derive optimal control for a nonlinear system. The fuzzy logic model is well-identified, accompanied by trial and error to tune parameters heuristically. In [35], the optimal control is designed within an actor–critic framework, where the extra neural network is employed to identify the unmodeled dynamic, incurring a heavy computation burden. Thus, some scholars have focused on critic-only ADP with less computation complexity. The critic NN is adopted to update optimal cost function and optimal policy for nonlinear known systems simultaneously [36]. Subsequently, a series of ADP results are reported for partially known nonlinear systems, where the robustness of a closed-loop system can be ensured by game theory [37,38], adding robust terms [39], sliding mode control [40], or DO [41]. Specifically, DO-based ADP is a mainstream solution to achieve robustness and optimal performance [42,43,44,45]. On the condition that the derivative of disturbance is close to zero, the optimal tracking control is considered in [42] by integrating improved DO with dual network ADP. In [43], by introducing finite-time DO with known upper bound information, an optimal sliding mode controller is designed based on actor–critic ADP. In [44], by combining with DO, the actor–critic ADP is employed to derive the optimal control of sliding-mode dynamics. The steady control input is put forward based on novel DO, and optimal control is derived for optimal regulation problems in a single-network ADP in [45].

It would be worth mentioning that the above-involved DOs were developed under the assumption that the upper bound of disturbance is available or its derivative tends to zero, which are troublesome and unsuitable for dealing with various kinds of disturbances in practical implementations. In addition, only the asymptotic or finite-time convergence of disturbance estimation errors can be achieved in the existing whole ADP control scheme, and hysteretic disturbance estimation may induce poor tracking performance. Note that fixed-time DO can enable disturbance estimation errors to converge to zero in a preassigned time in spite of initial estimation errors, but previously reported fixed-time DOs are established on the condition that the upper bound of disturbance is available. It is a challenge to estimate lumped disturbances for identifying a system without the assumption of bounded information on lumped disturbance. Moreover, the critic NN is conducted to approximate the optimal cost function and optimal policy to derive optimal control without actor NN.

Motivated by the previous discussion, a novel adaptive fixed-time disturbance observer (AFXDO)-based approximate optimal tracking control architecture for nonlinear partially unknown systems is considered, where AFXDO is designed such that the estimation of perturbation can be accomplished within a fixed time. Then, approximate optimal control is conducted by incorporating the real-time estimation of AFXDO into a critic-only ADP framework, where weight law with variable learning rate is embedded in a single network. The outstanding features of the devised scheme are:

(1) Compared with the presenting DO, the stability of perturbation estimation error can be guaranteed with finite-time convergence [24,25,26,27,28] or asymptotic convergence [21,22,23], and the investigated AFXDO can achieve the fixed-time stability of the estimation dynamic with the core of fixed-time stability system, ensuring that estimation error with fixed-time convergence with one power term. In addition, in view of the mentioned disturbance observer with a strong assumption about perturbation, the adaptive rule is employed to adjust observer gain, removing the rigid hypothesis on the known boundedness of perturbation [21,22,23,24].

(2) In contrast to the existing ADP framework with the actor–critic framework [42,43,44], the proposed ADP is a much simpler structure without using actor NN to estimate optimal policy. Different from the critic-only ADP framework with known upper-bounded information where the Bellman error is minimized by gradient descent [39,40], herein, by adaptive adjusting observer gain to estimate the perturbance, a controller is set up to eliminate the impact of perturbance, while the optimal controller is proposed with critic-only ADP with the weight law driven by extracted weight error conducted via filter operation, where the variable learning rate is utilized to fulfill faster weight convergence without initial chattering phenomenon. A brief comparison between current controllers and the proposed scheme is summarized in

Table 1. Comparison between the existing controllers and proposed scheme.

Literature	Controller Design
	Anti-Interference Manner	Assumptions	Convergence Results	Optimal Manner	Weight Update
[22]	Integral DO	Known	Asymptotic convergence	Unconsidered	Unconsidered
[24,25,26,27,28]	Finite-time DO	Known	Finite-time convergence	Unconsidered	Unconsidered
[30,31,32]	Fixed-time DO	Known	Fixed-time convergence	Unconsidered	Unconsidered
[39]	Value function	Known	Unconsidered	Critic-only	Gradient Descent
[40]	Sliding mode control	Known	Unconsidered	Critic-only	Gradient Descent
[42,44]	DO	Known	Asymptotic convergence	Actor–critic	Gradient Descent
[43]	Finite-time DO	Known	Finite-time Convergence	Actor–critic	Gradient Descent
Our method	Fixed-time DO	Unknown	Fixed-time convergence	Critic-only	Improved gradient

.

The remainder of this research is organized as follows: The problem formulation and its preliminaries are described in Section 3. In Section 4, the novel fixed-time observer is designed, while the optimal control policy is put forward with improved adaptive law and the stability analysis of improved adaptive law and error system are discussed. The simulation cases are carried out in Section 5. In Section 6, we present some conclusions.

3. Problem Formulation and Preliminaries

3.1. Problem Formulation

The nonlinear multi-input multi-output (MIMO) system is described with uncertain dynamics and perturbance as follows:

$$\dot{\mathit{x}}=\mathit{f}(\mathit{x})+\mathit{g}(\mathit{x})\mathit{u}+\mathit{d}(t)$$

(1)

where $\mathit{x}\in {ℝ}^n$ denotes the state vector and $\mathit{u}\in {ℝ}^m$ is the control input. $\mathit{f}(\mathit{x})\in {ℝ}^n$ and $\mathit{g}(\mathit{x})\in {ℝ}^{n\times m}$ indicate the dynamic drift and control gain matrix. $\mathit{f}(\mathit{x})$ is assumed to be an uncertain smooth and nonlinear function, which can be decomposed as $\mathit{f}(\mathit{x})={\mathit{f}}_0(\mathit{x})+\Delta \mathit{f}(\mathit{x})$ with known ${\mathit{f}}_0(\mathit{x})$ and bounded uncertainty $\Delta \mathit{f}(\mathit{x})$. The time-varying disturbance $\mathit{d}(t)\in {ℝ}^n$ represents bounded perturbation. The desired tracking trajectory ${\mathit{x}}_{\mathit{d}}$ and its first derivate ${\dot{\mathit{x}}}_{\mathit{d}}$ are supposed to be bounded, which can conduce the system being controllable on a compact set [46].

To fulfill optimal tracking control for system (1) with reference command, the tracking error is defined as $\mathit{e}=\mathit{x}-{\mathit{x}}_{\mathit{d}}={[e_1,e_2,...,e_n]}^T$. Thus, the tracking error of the system can be obtained by

$$\dot{\mathit{e}}=\dot{\mathit{x}}-{\dot{\mathit{x}}}_{\mathit{d}}={\mathit{f}}_0(\mathit{x})+\mathit{g}(\mathit{x})\mathit{u}+\Delta \mathit{f}(\mathit{x})+\mathit{d}(t)-{\dot{\mathit{x}}}_{\mathit{d}}$$

(2)

For the purpose of deriving the tracking error accurately, the controller is decomposed into two parts: the feedforward controller and the optimal tracking controller $\mathit{u}={\mathit{u}}_f+{\mathit{u}}_q$. The feedforward controller ${\mathit{u}}_f={[{\mathit{g}}^T\mathit{g}]}^{-1}{\mathit{g}}^T({\dot{\mathit{x}}}_d-{\mathit{f}}_0-\mathit{K}\mathit{e})$ with a positive-definite diagonal matrix $\mathit{K}\in {ℝ}^{n\times n}$ is developed by compensating the reference command and known dynamic drift ${\mathit{f}}_0(\mathit{x})$ to reduce the region of tracking error. Thus, by substituting ${\mathit{u}}_f$ into (2), the dynamic of the tracking error can be converted into

$$\dot{\mathit{e}}=\mathbf{-}\mathit{K}\mathit{e}+\mathit{g}(\mathit{x}){\mathit{u}}_q+\Delta \mathit{f}(\mathit{x})+\mathit{d}(t)$$

(3)

The control objective of this paper is to develop the optimal control policy ${\mathit{u}}_q$ for a nonlinear system in despite of perturbation and uncertain mode uniformly, such that the following goals are achieved:

(1)

The disturbance estimation error can converge to zero in a fixed time.

(2)

The trajectory tracking error and weight estimation error are uniformly ultimately bounded (UUB).

(3)

The designed control input approximates the optimal control policy.

Remark 1. 

In essence, there exist numerous nonlinear systems with disturbance and uncertainty, which can be expressed in the form of nonlinear MIMO systems (1) or high-order versions, such as robots [1], quadrotors [4], and MEMS [47], whereassumptions of the boundedness of the uncertainty and disturbance are acceptable. For the sake of analysis, we mainly focus on (1) as a first-order system, although the high-order system can be investigated via the proposed controller scheme incorporated with the backstepping technique. Different from the vast controllers explored in former works [22,23,24,25,26,27,28,29,30,31,32], where controllers lean on Lyapunov stability in the traditional framework, the investigated controller is based on the ADP scheme. Noting the uncertain term and disturbance in the system (3), the optimal tracking control will be clarified with AFXDO.

3.2. Related Lemmas

Lemma 1 

([32]). Suppose that there exists a positive function $V$satisfying

$$\dot{V}(t)\le -\ell V^{\chi }(t)-\iota V^{\alpha }(t)$$

(4)

where $\chi =\frac{\alpha }{2}+\frac{\beta }{2}+\left(\frac{\alpha }{2}-\frac{\beta }{2}\right)sign\left(V\left(t\right)-1\right)$ with $0<\beta <1$ and $\alpha >1$ and positive constants $\ell$ and $\iota$, then $V\left(t\right)$ can converge to the origin before the settling time:

$$T=\frac{1}{\ell \left(1-\beta \right)}+\frac{1}{\left(\ell +\iota \right)\left(\alpha -1\right)}$$

(5)

Lemma 2 

([48]). For positive number $\xi$ and positive integer $\mathrm{K}$, there holds

$${\displaystyle \sum _{k=1}^{\mathrm{K}}{\xi }^m}\ge {\left({\displaystyle \sum _{k=1}^{\mathrm{K}}\xi }\right)}^m, \mathrm{for} 0<m\le 1$$

(6)

$${\displaystyle \sum _{k=1}^{\mathrm{K}}{\xi }^m}\ge {\mathrm{K}}^{1-m}{\left({\displaystyle \sum _{k=1}^{\mathbf{K}}\xi }\right)}^m, \mathrm{for} m>1$$

(7)

Lemma 3  

([49]). For positive constants $a$ and $b$ with $p>1$ satisfying $\frac{1}{p}+\frac{1}{q}=1$, there holds

$$ab\le \frac{\tau a^p}{p}+\frac{b^q}{\tau q}$$

(8)

with positive $\tau$.

4. Main Results

In this section, the optimal tracking control based on AFXDO is developed to fulfill the specified goals in the above section. For the controller ${\mathit{u}}_q$, the novel fixed-time observer is ingeniously investigated to estimate the synthesized unknown term with adaptive observer gains. By virtue of the designed observer, the optimal control is derived with a critic-only ADP scheme, where an improved adaptive law with a variable learning rate is established for weight update. The flowchart of the proposed controller can be described in Figure 1.

4.1. Adaptive Fixed-Time Disturbance Observer

Inspired by [20], a novel AFXDO is developed without prior information of disturbance, ensuring that the estimation error converges to zero in a fixed time by virtue of adaptive observer gains. Noting that the uncertain dynamic drift and external disturbance are unknown, the AFXDO is conducted to estimate the synthesized unknown term as lumped disturbances:

$$\Delta =\Delta \mathit{f}(\mathit{x})+\mathit{d}(t)$$

(9)

Thus, the system (3) is transformed into $\dot{\mathit{e}}=\mathit{g}(\mathit{x}){\mathit{u}}_q+\Delta =\mathit{G}(\mathit{x},{\mathit{u}}_q)={\left[G_1,G_2,...,G_n\right]}^T$, and the AFXDO is devised for each component of tracking error and lumped disturbances as

$$\left\{\begin{array}{l}{\dot{\widehat{e}}}_i=k_1{\tilde{e}}_i^{\chi }\mathrm{sign}\left({\tilde{e}}_i\right)+G_i+{\widehat{\Delta }}_i+{\lambda }_1{\hslash }_i^{}\\ {\dot{\widehat{\Delta }}}_i=k_2{\hslash }_i^{\chi }\mathrm{sign}\left({\hslash }_i^{\chi }\right)+{\lambda }_2\mathrm{sign}\left({\hslash }_i^{\chi }\right)\end{array}\right.$$

(10)

where $\widehat{\mathit{e}}={[{\widehat{e}}_1,{\widehat{e}}_2,...,{\widehat{e}}_n]}^T$ and $\widehat{\Delta }={[{\Delta }_1,{\Delta }_2,...,{\Delta }_n]}^T$ represent the estimation of tracking error $\mathit{e}$ and synthesized uncertainties $\Delta$, ${\tilde{e}}_i=e_i-{\widehat{e}}_i$,${\hslash }_i^{}=r\mathrm{sign}\left({\tilde{e}}_i\right)$ for $i=1,2,...,n$ with positive constant $r$. The parameters $k_1$ and $k_2$ are positive parameters, $\chi =\frac{\alpha }{2}+\frac{\beta }{2}+\left(\frac{\alpha }{2}-\frac{\beta }{2}\right)\mathrm{sign}\left(\left|{\tilde{e}}_i\right|-1\right)$ with $\alpha >1$ and $0<\beta <\frac{1}{2}$. The observer gains ${\lambda }_1$ and ${\lambda }_2$ are adapted from following rules:

$$\left\{\begin{array}{l}{\dot{\lambda }}_1=l_1{\left|{\tilde{e}}_i\right|}^{1-2\beta }+{\mu }_1\\ {\dot{\lambda }}_2=l_2{\left|{\left({\hslash }_i\right)}_{eq}\right|}^{1-2\beta }+{\mu }_2\end{array}\right.$$

(11)

with positive parameters $l_1$, $l_2$, ${\mu }_1$, and ${\mu }_2$ to be chosen and ${\left({\hslash }_i\right)}_{eq}=r\mathrm{sign}\left({\tilde{e}}_i\right)$. Theorem 1 is given for disturbance estimation error, which is proved in Appendix A.

Theorem 1. 

Considering the system (1) and novel AFXDO (10) with adaptive gain update (11), the disturbance estimation error can converge to zero in a fixed time.

In view of the tracking error system (3), the feedback controller is set as follows:

$${\mathit{u}}_q=-{[{\mathit{g}}^T\mathit{g}]}^{-1}{\mathit{g}}^T\widehat{\Delta }+{\mathit{v}}_q$$

(12)

where $-{[{\mathit{g}}^T\mathit{g}]}^{-1}{\mathit{g}}^T\widehat{\Delta }$ is employed to attenuate the impression of uncertainty and the extra term is the optimal control policy of the following system:

$$\dot{\mathit{e}}=-\mathit{K}\mathit{e}+\mathit{g}(\mathit{e}){\mathit{v}}_{\mathit{q}}+\tilde{\Delta }$$

(13)

Remark 2. 

The settling time is only related to the control parameters and does not vary along with the initial condition, which dominates superiority among other DOs with finite-time convergence [24,25,26,27,28] and asymptotic convergence [20,21,22,23]. In addition, there is one power term in the observer (10) rather than existing fixed-time DOs with two power terms [18,19,20] to achieve fixed-time convergence. Note that the fixed-time stable system in [32] can be shown in the following equation as

$$\dot{z}=-\left({\gamma }_1{\left|z\right|}^{\chi }+{\gamma }_2{\left|z\right|}^{\beta }\right)sign\left(z(t)\right)$$

(14)

with positive constants ${\gamma }_1$ and ${\gamma }_2$. Based on Lemma 1, we can derive settling time as

$$T=\frac{1}{{\gamma }_1\left(\alpha -1\right)}+\frac{1}{\left({\gamma }_1+{\gamma }_2\right)\left(1-\beta \right)}$$

(15)

The improved version of (14) consisting of only the power term can be represented with positive constant ${\gamma }_1{}^{\prime }$ as

$$\dot{z}=-{\gamma }_1{}^{\prime }{\left|z\right|}^{\chi }sign\left(z(t)\right)$$

(16)

which is the core of the designed observer. Moreover, the adaptive rule (11), which is simple and easily operated, can enable sufficiently large increases in ${\lambda }_1$ and ${\lambda }_2$, removing the strict assumptions on perturbations, such as the known bounded information of the perturbation or its derivative and invariant perturbation [42,43]. Noting that there exist many parameters involved in the AFXDO, we summarize the fundamental guidelines for adjusting the observer performance as follows: The parameters $k_1$,  $k_2$, ${\lambda }_1$, ${\lambda }_2$, and exponential $\chi$ related to $\alpha$, $\beta$ are key factors to guarantee the settling time and accuracy. Noting that the $k_1$ is the identical gain for ${\tilde{e}}_i>1$ and ${\tilde{e}}_i<1$, we should make a compromise on the convergence rate and accuracy. In addition, the adaptive rules (11) are adopted for increasing gains ${\lambda }_1$ and ${\lambda }_2$ with adequate parameters ${\mu }_1, {\mu }_2, l_1, l_2$ for observer stability.

Remark 3. 

Existing optimal controls based on the ADP scheme mainly concentrate on the optimal regulation problem, which is challenging for designing optimal tracking control, especially for uncertainty systems as well as nonvanishing tracking trajectories. The feedforward controller is conducted to compensate for the nonvanishing tracking trajectory and known drift dynamic. Thus, the optimal trajectory controller for system (1) is converted to an optimal regulation problem (3). For addressing the uncertainty, a novel AFXDO is ingeniously conducted for the estimation of lumped disturbances, where the disturbance estimation errors can converge to zero in a fixed time and observer gains are tuned adaptively without an assumption of the known upper bound of $\Delta$.

4.2. Optimal Tracking Control Scheme

The optimal control is considered to stabilize the system (13). Unlike the traditional controller in [29,30,31,32], there is only taking tracking performance into consideration without further exploring optimal control. A cost function is denoted related to tracking error and control input ${\mathit{v}}_q$ as

$$V(\mathit{e})={\displaystyle {\int }_t^{\infty }c\left(\mathit{e},{\mathit{v}}_{\mathit{q}}\right)d\varsigma }$$

(17)

where $c\left(\mathit{e},{\mathit{v}}_{\mathit{q}}\right)={\mathit{e}}^T\mathit{Q}\mathit{e}+{\mathit{v}}_{\mathit{q}}^T\mathit{R}{\mathit{v}}_{\mathit{q}}$ with positive-definite diagonal matrices $\mathit{Q}\in {ℝ}^{n\times n}$ and $\mathit{R}\in {ℝ}^{m\times m}$. One can denote the optimal cost function:

$$V*\left(\mathit{e}\right)=\underset{{\mathit{v}}_{\mathit{q}}\in \mathsf{\Psi }\left(\mathsf{\Omega }\right)}{\min }{\displaystyle {\int }_t^{\infty }c\left(\mathit{e},{\mathit{v}}_{\mathit{q}}^{*}\right)d\varsigma }$$

(18)

Taking the time derivative of (18) along with (13), one can obtain the HJB equation as

$$H\left(\mathit{e},{\mathit{v}}_{\mathit{q}}^{*},V^{*}\right)={\mathit{V}}_e^{*}{}^T\left[-\mathit{K}\mathit{e}+\mathit{g}{\mathit{v}}_{\mathit{q}}^{*}+\tilde{\Delta }\right]+{\mathit{e}}^{\mathit{T}}\mathit{Q}\mathit{e}+{\mathit{v}}_{\mathit{q}}^{*}{}_{}^T\mathit{R}{\mathit{v}}_{\mathit{q}}^{*}=\mathbf{0}$$

(19)

where ${\mathit{V}}_e^{*}=\frac{\partial V*\left(\mathit{e}\right)}{\partial \mathit{e}}$. According to (19), optimal control ${\mathit{v}}_{\mathit{q}}^{\mathbf{*}}$ can be conducted by $\frac{\partial H\left(\mathit{e},{\mathit{v}}_{\mathit{q}}^{\mathbf{*}},V*\right)}{\partial {\mathit{v}}_{\mathit{q}}^{\mathbf{*}}}=0$ as

$${\mathit{v}}_{\mathit{q}}^{\mathbf{*}}=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T{\mathit{V}}_e^{*}$$

(20)

Noting that the optimal control ${\mathit{v}}_{\mathit{q}}^{\mathit{*}}$ (20) depends on the gradient of the cost function, i.e., ${\mathit{V}}_e^{*}$, by substituting (20) into (19), the HJB Equation (19) can be represented as

$${\mathit{V}}_e^{*T}[\mathbf{-}\mathit{K}\mathit{e}+\tilde{\Delta }]+{\mathit{e}}^{\mathit{T}}\mathit{Q}\mathit{e}\mathbf{-}\frac{1}{4}{\mathit{V}}_e^{*T}\mathit{g}\mathit{R}{\mathit{g}}^T{\mathit{V}}_e^{*}=0$$

(21)

However, it is difficult and even impossible to achieve the analytical solution due to the nonlinearity of the HJB equation. For addressing this issue, an NN approximator named critic-only NN is introduced to approximate the optimal cost function $V*\left(\mathit{e}\right)$, and the optimal control policy can be achieved immediately rather than depending on another NN.

Specifically, the optimal cost function $V*(e)$ can be approximated by a single-layer NN, which can be expressed as

$$V*(\mathit{e})={\mathit{W}}^T\mathit{\sigma }(\mathit{e})+\epsilon$$

(22)

where $\mathit{W}\in {ℝ}^p$ represents the ideal weight of the NN, with $p$ indicating the number of neurons, $\mathit{\sigma }(\mathit{e})\in {ℝ}^p$ being the corresponding activation function, and $\epsilon$ being the approximate error induced by the critic NN. From (22), the first derivative of $V*(\mathit{e})$ with respect to tracking error can be obtained as

$$\frac{\partial V*(\mathit{e})}{\partial \mathit{e}}=\nabla \mathsf{\sigma }{(\mathit{e})}^T\mathit{W}+\nabla \mathsf{\epsilon }$$

(23)

where $\nabla \mathsf{\sigma }\left(\mathit{e}\right)=\frac{\partial \mathsf{\sigma }(e)}{\partial \mathit{e}}$ and $\nabla \mathsf{\epsilon }=\frac{\partial \epsilon }{\partial \mathit{e}}$ are the gradient of the activation function and approximation error with upper boundness $\overline{\sigma }$ and $\overline{\epsilon }$ satisfying $‖\nabla \mathsf{\sigma }‖\le \overline{\sigma }$ and $‖\nabla \mathsf{\epsilon }‖\le \overline{\epsilon }$. Thus, the optimal control ${\mathit{v}}_{\mathit{q}}^{\mathbf{*}}$ can be given as:

$${\mathit{v}}_{\mathit{q}}^{\mathbf{*}}=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T(\nabla \mathsf{\sigma }{(\mathit{e})}^T\mathit{W}+\nabla \mathsf{\epsilon })$$

(24)

Since the ideal weights that can accurately estimate the cost function are unavailable, its approximation can be estimated by estimation weight and corresponding regressors:

$$\widehat{V}(\mathit{e})={\widehat{\mathit{W}}}^T\mathsf{\sigma }(\mathit{e})$$

(25)

with $\widehat{\mathit{W}}$ representing an estimation of the ideal weight. The actual approximated optimal policy is induced by

$${\mathit{v}}_{\mathit{q}}^{}=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla \mathsf{\sigma }{(\mathit{e})}^T\widehat{\mathit{W}}$$

(26)

Then approximation HJB Equation (19) with (26) can be expressed as

$${\mathit{W}}^T\nabla \mathsf{\sigma }(-\mathit{K}\mathit{e}+\mathit{g}{\mathit{v}}_{\mathit{q}}^{})+{\mathit{e}}^{\mathit{T}}\mathit{Q}\mathit{e}+{\mathit{v}}_{\mathit{q}}^{\mathit{T}}\mathit{R}{\mathit{v}}_{\mathit{q}}^{}+{\epsilon }_{HJB}=0$$

(27)

with ${\epsilon }_{HJB}$ being the HJB error associated with NN approximation error and disturbance error and denoting ${\epsilon }_{HJB}=\nabla {\mathsf{\epsilon }}^T(-\mathit{K}\mathit{e}+\mathit{g}{\mathit{v}}_{\mathit{q}}^{}+\tilde{\Delta })+({\mathit{W}}^T\nabla \mathsf{\sigma })\tilde{\Delta }$. With sufficiently large $p$ and the fixed-time convergence of AFXDO, it can be guaranteed that $\nabla \mathsf{\epsilon }$ and $\tilde{\Delta }$ converge to zero.

Then, (27) can be simplified as a linearization form:

$$\Psi =-{\mathsf{\Phi }}^T\mathit{W}-{\epsilon }_{HJB}$$

(28)

with $\Psi ={\mathit{e}}^{\mathit{T}}\mathit{Q}\mathit{e}+{\mathit{v}}_{\mathit{q}}^{\mathit{T}}\mathit{R}{\mathit{v}}_{\mathit{q}}^{}$ and $\mathsf{\Phi }=\nabla \mathsf{\sigma }(-\mathit{K}\mathit{e}+\mathit{g}{\mathit{v}}_{\mathit{q}}^{})$. There have been many update laws for online updating weight, such as gradient descent and least squares methods [39,40] with fixed learning rates. Different from the Bellman error driving weight update in the inherent ADP schemes, the weight errors can be extracted by auxiliary operation, leading to the fact that weight errors can converge to the neighborhood of ideal weight and further enhance weight estimation performance—specifically, by introducing auxiliary matrix $\mathit{M}\in {ℝ}^{p\times p}$ and vector $\mathit{N}\in {ℝ}_{}^p$, which are defined as:

$$\left\{\begin{array}{c}\dot{\mathit{M}}=-c\mathit{M}+\mathsf{\Phi }{\mathsf{\Phi }}^T,\hspace{1em}\hspace{1em}\mathit{M}\left(0\right)=0\\ \dot{\mathit{N}}=-c\mathit{N}+\mathsf{\Phi }\Psi ,\hspace{1em}\hspace{1em}\hspace{1em}\mathit{N}\left(0\right)=0\end{array}\right.$$

(29)

with designed constant $c>0$. Denoting $\mathsf{\Im }=\mathit{M}\widehat{\mathit{W}}+\mathit{N}$, the estimation rule of weight in [50] can be employed for weight update as

$$\dot{\widehat{\mathit{W}}}=-\mathsf{\Upsilon }\mathsf{\Im }, \mathsf{\Upsilon }={\mathsf{\Upsilon }}_0{e}^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}$$

(30)

with positive-definite diagonal matrices $\mathsf{\Upsilon }\in {ℝ}^{p\times p}$ and ${\mathsf{\Upsilon }}_0\in {ℝ}^{p\times p}$ and constant $\delta >0$. The weight convergence can be guaranteed by following the theorem with ingenious derivation, in which there contains the weight error $\tilde{\mathit{W}}=\mathit{W}-\widehat{\mathit{W}}$ in (30).

Theorem 2. 

For the approximation of the cost function with critic NN (22), if the estimation cost function can be evolved by the estimation of weight with update rule (30), the weight errors can be driven into the neighborhood of zero.

Proof of Theorem 2. 

By integrating both sides of (29), the solution can be formulated as

$$\left\{\begin{array}{c}\mathit{M}(t)={\displaystyle {\int }_0^t{e}^{-c\left(t-r\right)}\mathsf{\Phi }{\mathsf{\Phi }}^{T}dr},\hspace{1em}\hspace{1em}\mathit{M}(0)=0\\ \mathit{N}(t)={\displaystyle {\int }_0^t{e}^{-c\left(t-r\right)}\mathbf{\Phi }{\Psi }^{T}dr,}\hspace{1em}\hspace{1em}\mathit{N}(0)=0\end{array}\right.$$

(31)

Combined with (45), one can derive that there exists an auxiliary variable $\mathsf{o}$ being associated to HJB error, i.e., $\mathsf{o}=-{\displaystyle {\int }_0^t{e}^{-c\left(t-r\right)}\mathsf{\Phi }{\epsilon }_{HJB}dr}$, which satisfies $\mathit{N}=-\mathit{M}\mathit{W}+\mathsf{o}$. Following (30), this can be a further representation of the update rule:

$$\mathsf{\Im }=\mathit{M}\widehat{\mathit{W}}+\mathit{N}=-\mathit{M}\tilde{\mathit{W}}+\mathsf{o}$$

(32)

where there exists a positive constant ${\epsilon }_{\mathit{o}}$, such that $‖\mathsf{o}‖\le {\epsilon }_{\mathit{o}}$ holds. Obviously, the weight errors can be extracted via the update rule, leading to weight convergence. Under the condition that $\mathsf{\Phi }$ satisfies the persistent excitation (PE) condition, the positive-definite matrix $\mathit{M}$ can be derived from Lemma 1 in [50]. Consider the following Lyapunov function:

$$V_W=\frac{1}{2}{\tilde{\mathit{W}}}^T{\mathsf{\Upsilon }}_0^{{}^{-1}}\tilde{\mathit{W}}$$

(33)

According to (30), one can derive the time derivative of $V_W$, such that:

$$\begin{array}{ll}{\dot{V}}_W& ={\tilde{\mathit{W}}}^T{\mathsf{\Upsilon }}_0^{{}^{-1}}\dot{\tilde{\mathit{W}}}=e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}{\tilde{\mathit{W}}}^T\left(-\mathit{M}\tilde{\mathit{W}}+\mathsf{o}\right)\\ & \le e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}\left(-\eta {‖\tilde{\mathit{W}}‖}^2+‖\tilde{\mathit{W}}‖‖\mathsf{o}‖\right)\\ & \le -e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}‖\tilde{\mathit{W}}‖\left(\eta ‖\tilde{\mathit{W}}‖-{\epsilon }_{\mathit{o}}\right)\end{array}$$

(34)

with $\eta >0$ denoting the minimized eigenvalue of the matrix, i.e., $\eta ={\lambda }_{\min }\left(M\right)$. By revisiting the Lyapunov theory, the weight errors can converge to the neighborhood of origin. □

Remark 4. 

Contrary to most ADP schemes, where the Hamiltonian function is structured for the nominal system, the HJB equation is conducted related to (13), consisting of the estimation error, which leads to challenges for adjusting the optimal control policy and weight update. As opposed to the existing control within the ADP framework, where the weight update is carried out by minimizing the Bellman error with gradient descent [39,40], by imposing an auxiliary operation on intermediate parameters and employing the Gaussian function to tune the learning rate, the adaptive rule (30), which contains the weight errors, can compel the approximation of weight converge to an ideal weight. In particular, when the tracking error is far from the origin, $\mathsf{\Upsilon }$ is decreased to avoid the initial oscillation of weight, which further affects the control input. If the tracking error is close to zero, larger $\mathsf{\Upsilon }$ can enable online learning ability with an acceleration convergence of weight, and the PE condition can be easily verified by transferring it into the positive-definite property of matrix $\mathit{N}$, avoiding introducing the concurrent learning or the experience replay employed in most ADP frameworks.

4.3. Stability Analysis

Theorem 3. 

For the partially known nonlinear system (3) with uncertain parameters and unknown disturbance, by designing control (26) with weight updating laws (30), the following goals can be achieved.

(1) 

The tracking error and weight error are UUB.

(2) 

The approximate actual control input approximates the optimal control policy.

Proof of Theorem 3. 

According to (13) and (26), there exist dynamics of tracking errors such as:

$$\begin{array}{ll}\dot{\mathit{e}}& =-\mathit{K}\mathit{e}+\mathit{g}{\mathit{v}}_{\mathit{q}}^{}+\tilde{\Delta }\\ & =-\mathit{K}\mathit{e}-\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\widehat{\mathit{W}}+\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\left(\nabla {\mathsf{\sigma }}^T\mathit{W}+\nabla \mathsf{\epsilon }\right)+\mathit{g}{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}+\tilde{\Delta }\\ & =-\mathit{K}\mathit{e}+\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\tilde{\mathit{W}}+\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla \mathsf{\epsilon }+\mathit{g}{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}+\tilde{\Delta }\end{array}$$

(35)

Thus, the following Lyapunov function is constructed for proving the stability of tracking errors:

$$V=\frac{1}{2}{\tilde{\mathit{W}}}^T{\mathsf{\Upsilon }}_0^{{}^{-1}}\tilde{\mathit{W}}+{\Gamma }_1{\mathit{e}}^{\mathit{T}}\mathit{e}+{\Gamma }_{2}V*+{\Gamma }_3{\mathsf{o}}^T\mathsf{o}$$

(36)

with optimal function $V*$ denoted by (18) and positive constants ${\Gamma }_1, {\Gamma }_2, {\Gamma }_3$. In the following proof, each individual part of (36) is concretely analyzed based on Lemma 3.

First, we restate the convergence of $\tilde{\mathit{W}}$ as follows:

$$\begin{array}{ll}{\dot{V}}_1& =e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}{\tilde{\mathit{W}}}^{\mathit{T}}\left(-\mathit{M}\tilde{\mathit{W}}+\mathsf{o}\right)\\ & \le -e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}\left(\eta {‖\tilde{\mathit{W}}‖}^2+‖{\tilde{\mathit{W}}}^T\mathsf{o}‖\right)\\ & =-e^{-\frac{{‖e‖}^2}{{\delta }^2}}\left[\left(\eta -\frac{1}{2\tau {\Gamma }_3}\right){‖\tilde{\mathit{W}}‖}^2-\frac{\tau {\Gamma }_3{‖\mathsf{o}‖}^2}{2}\right]\end{array}$$

(37)

Thus, by substituting (35) into (36), it can be further reformulated in a simple form:

$$\begin{array}{ll}{\dot{V}}_2& =2{\Gamma }_1{\mathit{e}}^{\mathit{T}}\dot{\mathit{e}}+{\Gamma }_2{\dot{V}}^{*}\\ & =2{\Gamma }_1{\mathit{e}}^{\mathit{T}}\dot{\mathit{e}}+{\Gamma }_2(-{\mathit{e}}^T\mathit{Q}\mathit{e}-{\mathit{v}}_{\mathit{q}}^{\mathbf{*}\mathit{T}}\mathit{R}{\mathit{v}}_{\mathit{q}}^{\mathbf{*}})\\ & =2{\Gamma }_1{\mathit{e}}^T\left[-{\lambda }_{\min }(\mathit{K})\mathit{e}+\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\tilde{\mathit{W}}+\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla \mathsf{\epsilon }+\mathit{g}{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}+\tilde{\Delta }\right]\\ & +{\Gamma }_2(-{\mathit{e}}^T\mathit{Q}\mathit{e}-{\mathit{v}}_{\mathit{q}}^{\mathbf{*}\mathit{T}}\mathit{R}{\mathit{v}}_{\mathit{q}}^{*})\\ & \le -\left[2{\lambda }_{\min }(\mathit{K}){\Gamma }_1+{\Gamma }_2{\lambda }_{\min }(\mathit{Q})-\left(‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T‖+‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T‖+2\right){\Gamma }_1\right]{‖\mathit{e}‖}^2\\ & +\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T‖{‖\tilde{\mathit{W}}‖}^2+\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T‖{‖\nabla \mathsf{\epsilon }‖}^2+{\Gamma }_1{‖\tilde{\Delta }‖}^2\\ & -\left({\Gamma }_2{\lambda }_{\min }(\mathit{R})-{\Gamma }_1{‖\mathit{g}‖}^2\right){‖{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}‖}^2\end{array}$$

(38)

It can be derived from (31) that $\dot{\mathsf{o}}=-c\mathsf{o}+\mathsf{\Phi }{\epsilon }_{HJB}$, so that

$$\begin{array}{ll}{\dot{V}}_3& =2{\Gamma }_3{\mathsf{o}}^T\dot{\mathsf{o}}\\ & =2{\Gamma }_3{\mathsf{o}}^T\left(-c\mathsf{o}+\mathsf{\Phi }{\epsilon }_{HJB}\right)\\ & =2{\Gamma }_3{\mathsf{o}}^T\left\{-c\mathsf{o}+\mathsf{\Phi }\left[\left({\mathit{W}}^T\nabla \sigma +\nabla \mathsf{\epsilon }\right)\tilde{\Delta }+\nabla \mathsf{\epsilon }\left(-\frac{1}{2}\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\widehat{\mathit{W}}\right)-\nabla \mathsf{\epsilon }\mathit{K}\mathit{e}\right]\right\}\\ & =-{\Gamma }_3\left(2c-3\tau \right){‖\mathsf{o}‖}^2+\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\left({\mathit{W}}^T\nabla \mathsf{\sigma }+\nabla \mathsf{\epsilon }\right)‖}^2{‖\tilde{\Delta }‖}^2+\frac{{\Gamma }_3}{4\tau }{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T\widehat{\mathit{W}}‖}^2\\ & +\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{K}‖}^2{‖\mathit{e}‖}^2\end{array}$$

(39)

Therefore,

$$\begin{array}{ll}\dot{V}& ={\dot{V}}_1+{\dot{V}}_2+{\dot{V}}_3\\ & \le -\left[e^{-\frac{{‖e‖}^2}{{\delta }^2}}\left(\eta -\frac{1}{2\tau {\Gamma }_3}\right)-\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖\right]{‖\tilde{\mathit{W}}‖}^2-\left(K^{\prime }{\lambda }_{\min }\left(\mathit{R}\right)-{\Gamma }^{\prime }{‖\mathit{g}‖}^2\right){‖{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}‖}^2\\ & \hspace{1em}-\left[2{\lambda }_{\min }\left(\mathit{K}\right){\Gamma }_1+{\Gamma }_2{\lambda }_{\min }\left(\mathit{Q}\right)-\left(‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖+‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖+2\right){\Gamma }_1-\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{K}‖}^2\right]{‖\mathit{e}‖}^2\\ & \hspace{1em}-{\Gamma }_3\left(2c+\left(\frac{1}{2}{e}^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}-3\right)\tau \right){‖\mathsf{o}‖}^2+\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖{‖\nabla \mathsf{\epsilon }‖}^2+\frac{{\Gamma }_3}{4\tau }{‖\mathsf{\Phi }\nabla \epsilon \mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T\widehat{\mathit{W}}‖}^2\\ & \hspace{1em}+\left({\Gamma }_1+\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\left({\mathit{W}}^T\nabla \mathsf{\sigma }+\nabla \mathsf{\epsilon }\right)‖}^2\right){‖\tilde{\Delta }‖}^2\end{array}$$

(40)

As long as ${\Gamma }_2{\lambda }_{\min }\left(\mathit{R}\right)-{\Gamma }_1{‖\mathit{g}‖}^2>0$, (40) can be rewritten as

$$\dot{V}\le -c_1{‖\tilde{\mathit{W}}‖}^2-c_2{‖\mathit{e}‖}^2-c_3{‖\mathsf{o}‖}^2+\rho$$

(41)

with positive coefficients

$$c_1=e^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}\left(\eta -\frac{1}{2\tau {\Gamma }_3}\right)-\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖$$

$$c_2=2{\lambda }_{\min }\left(\mathit{K}\right){\Gamma }_1+{\Gamma }_2{\lambda }_{\min }\left(\mathit{Q}\right)-\left(‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖+‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖+2\right){\Gamma }_1-\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{K}‖}^2$$

$$c_3={\Gamma }_3\left(2c+\left(\frac{1}{2}{e}^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}-3\right)\tau \right)$$

$$\rho =\frac{1}{4}{\Gamma }_1‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖{‖\nabla \mathsf{\epsilon }‖}^2+\frac{{\Gamma }_3}{4\tau }{‖\mathsf{\Phi }\nabla \epsilon \mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\sigma }^T\widehat{\mathit{W}}‖}^2+\left({\Gamma }_1+\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\left({\mathit{W}}^T\nabla \mathsf{\sigma }+\nabla \mathsf{\epsilon }\right)‖}^2\right){‖\tilde{\Delta }‖}^2$$

Obviously, the upper bound of $\rho$ can be easily guaranteed from the NN approximation errors and HJB errors, while the observer error $\tilde{\Delta }$ will converge to zero in a fixed time. If suitable parameters are selected to ensure the positiveness of $c_1$, $c_2$, and $c_3$, it must satisfy the following conditions

$${\Gamma }_1<\frac{4\eta }{‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖}$$

$$\tau >\max \left(\frac{1}{{\Gamma }_3\left(2\eta -\frac{1}{2}{\Gamma }_2‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖\right)},\frac{{\Gamma }_3{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{K}‖}^2}{2{\lambda }_{\min }\left(\mathit{K}\right){\Gamma }_2}\right)$$

$${\Gamma }_2>\max \left(\frac{-2{\lambda }_{\min }\left(\mathit{K}\right){\Gamma }_1+\frac{{\Gamma }_3}{\tau }{‖\mathsf{\Phi }\nabla \mathsf{\epsilon }\mathit{K}‖}^2-\left(‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖+‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖+2\right){\Gamma }_1}{{\lambda }_{\min }\left(\mathit{Q}\right)},\frac{{\Gamma }_1{‖\mathit{g}‖}^2}{{\lambda }_{\min }\left(\mathit{R}\right)}\right)$$

$${\lambda }_{\min }\left(\mathit{K}\right)>\frac{‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla {\mathsf{\sigma }}^T‖+‖\mathit{g}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}‖}{2}+1$$

$$c>\left(-\frac{1}{4}{e}^{-\frac{{‖\mathit{e}‖}^2}{{\delta }^2}}+\frac{3}{2}\right)\tau >\frac{5}{4}\tau , {\Gamma }_3>0.$$

Noting that critic NN errors can be bounded with finite constants, i.e., $\rho >0$, it is obviously derived that $\dot{V}<0$ as long as one of the following inequalities holds:

$$‖\tilde{\mathit{W}}‖>\sqrt{\frac{\rho }{c_1}}, ‖\mathit{e}‖>\sqrt{\frac{\rho }{c_2}}, ‖\mathsf{o}‖>\sqrt{\frac{\rho }{c_3}},$$

(42)

Furthermore, tracking error $e$ and NN weights error $\tilde{\mathit{W}}$ are UUB. From (24) and (26), one can infer that

$$\begin{array}{ll}\underset{t\to \infty }{\lim }‖{\mathit{v}}_{\mathit{q}}^{}-{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}‖& =\underset{t\to \infty }{\lim }‖\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\tilde{\mathit{W}}+\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^{\mathit{T}}\nabla \mathsf{\epsilon }‖\\ & \le \frac{1}{2}‖{\mathit{R}}^{-1}{\mathit{g}}^T‖\left(\overline{\sigma }‖\tilde{\mathit{W}}‖+\overline{\epsilon }\right)\\ & \triangleq {\epsilon }_u.\end{array}$$

(43)

Especially, if the critic NN error is zero, i.e., $\nabla \mathsf{\epsilon }=0$, we can obtain $\rho =0$, resulting in

$$\dot{V}=-c_1{‖\tilde{\mathit{W}}‖}^2-c_2{‖\mathit{e}‖}^2\le 0$$

(44)

We can obtain $V\to 0$ from the Lyapunov Theorem, leading to $\tilde{\mathit{W}}$ and $\mathit{e}$ converging to zero. Thus, the actual control input approach to the optimal control policy is as follows:

$$\begin{array}{ll}\underset{t\to \infty }{\lim }‖{\mathit{v}}_{\mathit{q}}^{}-{\mathit{v}}_{\mathit{q}}^{\mathbf{*}}‖& =\underset{t\to \infty }{\lim }‖\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla {\mathsf{\sigma }}^T\tilde{\mathit{W}}+\frac{1}{2}{\mathit{R}}^{-1}{\mathit{g}}^T\nabla \mathsf{\epsilon }‖\\ & \le \frac{\overline{\sigma }}{2}‖{\mathit{R}}^{-1}{\mathit{g}}^T‖‖\tilde{\mathit{W}}‖\\ & =0.\end{array}$$

(45)

□

5. Simulation and Discussion

We have simulated a typical disturbance nonlinear system and confirmed the availability and superiority of the optimal tracking control based on disturbance compensation. Whether it is a quadrotor or a UAV operating with a specific task, there is a complex nonlinear partially known system, which induces challenges for control design to realize tracking control. Many scholars have paid attention to this problem and achieved productive research [4,6]. The motion of the quadrotor can be described by the rigid body model, where the Euclidean position in an inertia frame and Euler angle in a body-fixed frame are expressed by $\mathit{p}={[p_x,p_y,p_z]}^T$ and $\mathit{v}=\dot{\mathit{p}}={[v_x,v_y,v_z]}^T$. Noting the rotation matrix

$$\mathbf{H}=\left[\begin{array}{ccc}\cos \theta \cos \psi & \cos \theta \sin \varphi \cos \psi -\cos \varphi \sin \psi & \sin \psi \sin \theta +\sin \theta \cos \psi \cos \phi \\ \cos \theta \sin \psi & \cos \varphi \cos \psi +\sin \varphi \sin \theta \cos \psi & \sin \theta \cos \varphi \cos \psi -\cos \psi \sin \varphi \\ -\sin \theta & \cos \theta \sin \varphi & \cos \theta \cos \varphi \end{array}\right]\in {ℝ}^{3\times 3}$$

(46)

with $\phi ,\theta ,\psi$ representing roll, pitch, and yaw angles, and leaving out vane fluttering and air-operated friction, the dynamics of a quadrotor can be modeled as in the following formulae,

$$\begin{array}{l} \dot{\mathit{p}}=\mathit{v}\\ m\dot{\mathit{v}}=-mg{\mathit{h}}_{\mathit{3}}+H\mathit{u}-\mathbf{\prod }\mathit{v}+\mathsf{\wp }\end{array}$$

(47)

with $m$ and $g$ being mass and gravitational acceleration, ${\mathit{h}}_{\mathit{3}}={[0,0,1]}^T$. $\mathbf{\prod }$ and $\wp$ are unknown bounded uncertainties. To employ the designed controller, (47) is transferred into the following compact form as

$$\dot{\mathit{X}}={\mathit{f}}_0\left(\mathit{X}\right)+\mathit{g}(\mathit{X})\mathit{u}+\Delta \mathit{f}\left(\mathit{X}\right)+\mathit{d}(t)$$

(48)

where

$$\begin{gathered} \mathit{X}=\left[\begin{array}{l}\mathit{p}\\ \mathit{v}\end{array}\right]\in {ℝ}^{6\times 1}, {\mathit{f}}_0\left(\mathit{X}\right)=\left[\begin{array}{c}\mathit{v}\\ -g{\mathit{h}}_{\mathit{3}}\end{array}\right]\in {ℝ}^{6\times 1}, \mathit{g}(\mathit{X})=\left[\begin{array}{l}\mathit{0}\\ \mathit{I}\end{array}\right]\in {ℝ}^{6\times 3} \\ \mathit{u}=\left[\begin{array}{l}{u}_x\\ u_y\\ u_z\end{array}\right]\in {ℝ}^{3\times 1}, \Delta \mathit{f}\left(\mathit{X}\right)=\left[\begin{array}{c}\mathit{0}\\ \mathit{-}\mathsf{\prod }\mathit{v}\end{array}\right]\in {ℝ}^{6\times 1}, \mathit{d}(t)=\left[\begin{array}{l}\mathit{0}\\ \mathsf{\wp }/m\end{array}\right]\in {ℝ}^{6\times 1} \end{gathered}$$

(49)

Denoting the synthesized unknown term as stated in (9), i.e., $\Delta =\Delta \mathit{f}(\mathit{x})+\mathit{d}(t)$, the system parameters and reference command of position and velocity are listed in

Table 2. Model parameters and reference commands.

Variable	Value
$m$	$2\mathrm{kg}$
$g$	$9.8 {\mathrm{m/s}}^2$
$\mathbf{\prod }$	$\mathrm{diag}\left\{0.01,0.01,0.01\right\}{\mathrm{Nms}}^2$
$\mathit{d}$	${[\sin (4t)+\cos (2t)-\sin (t);\cos (4t)+\sin (2t)-\cos (t);\sin (3t)\cos (2t)-\cos (t)]}^T$
${\mathit{p}}_d^{}$	${[10(1-\cos (0.1\pi t)),5\sin (0.2\pi t),9(1-e^{-0.3t})]}^T$
${\mathit{v}}_d$	${[\pi \sin (0.1\pi t),\pi \cos \left(0.2\pi t\right),2.7e^{-0.3t}]}^T$

. The initial position and velocity are $\mathit{p}(0)={[-5,3,2]}^T$, $\mathit{v}(0)={[0,0,0]}^T$.

On account of the mentioned controller design, the devised controller is based on AFXDO and optimal control policy, where the unknown term is estimated by AFXDO. The optimal control policy is designed based on the ADP framework with the novel adaptive law (30). The optimal cost function is approximated by critic NN, in which Gaussian basis functions $\mathsf{\sigma }={[{\sigma }_1,{\sigma }_2,...,{\sigma }_{15}]}^T$ are considered regressors

$${\sigma }_i=\exp [\frac{-{‖\mathit{e}-{\mathit{a}}_{\mathit{i}}‖}^2}{q_i^2}],\hspace{0.33em}i=1,2,...,15$$

(50)

with $\mathit{e}={[{\mathit{p}}_{}^T-{\mathit{p}}_d^T\mathbf{,}{\mathit{v}}^T-{\mathit{v}}_d^T]}^T$, the width of the Gaussian function is set as $q_i=5$, and the centers ${\mathit{a}}_{\mathit{i}}={[a_{i1,}{a}_{i2,}...,a_{i6}]}^T$ are evenly distributed over the state space $[-5,5]$. Correspondingly, we choose the initial weight $\mathit{W}(0)={[0,0,...,0]}^T$$\in {ℝ}^{15}$ for updating critic NN weights $\mathit{W}={[W_1,W_2,...,W_{15}]}^T\in {ℝ}^{15}$. The computer programming is clearly clarified in

Table 3. The computer programming of the proposed scheme.

Step 1	Initialize $W=0,\mathit{M}=0,\hspace{0.33em}\mathit{N}=0$.
Step 2	Compute the tracking errors, feedforward controller ${\mathit{u}}_f$ and lumped disturbance approximation (10).
Step 3	Compute approximation of cost function (25) with critic NN.
Step 4	Update the auxiliary variables $\mathit{M}$ and $\mathit{N}$(29).
Step 5	Update the weight of critic NN (30).
Step 6	Calculate the optimal control ${\mathit{v}}_{\mathit{q}}$(26) with the approximate weight of critic NN.
Step 7	Update the controller ${\mathit{u}}_{\mathit{q}}$ according to lumped disturbance approximation and optimal control.
Step 8	If $t<t_f$, return to Step 2; else stop.

with specific steps. We will verify the validity of the proposed scheme from the observer and weight law.

• Case 1: Effect of disturbance observer on closed-loop performance

To verify the superiority of the proposed AFXDO, the following controllers are carried out, where different observers are inserted into the proposed ADP controller. For making a fair comparison, the controller parameters are set as: $K=\mathrm{diag}[1,1,1,1,1,1]$, $\mathit{Q}=\mathit{I}$, $\mathit{R}=\mathit{I}$,${\mathsf{\Upsilon }}_0=100\mathit{I}$, and $\delta =2.1$.

(1)

The designed ADP controller with extended state observer (ESO-ADP): The observer in [4] is utilized for the estimation of disturbance with the ADP framework. The bandwidth is set as $w_0=8$.

(2)

The designed ADP controller with finite-time DO (FIDO-ADP): The finite-time DO in [28] is employed with adaptive gains to eliminate the effect of disturbance in a finite time. The parameters of the observer are set as $\beta =5/11$, $k_1=k_2=10$, $l_1=l_2=0.01$, and ${\mu }_1={\mu }_2=0.001$.

(3)

The designed ADP controller with the proposed fixed-time DO (FXDO-ADP): During the operation of this method, we select the following parameters in AFXDO to ensure stability: $\alpha =2$, $\beta =5/11$, $k_1=k_2=10$, $l_1=l_2=0.01$, and ${\mu }_1={\mu }_2=0.001$.

It is worth noting that the different observers are employed in the critic-only ADP scheme in the above algorithm. Based on the structure of the observer and the ADP framework, the time and space complexity are consistent. The simulation results are illustrated in Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6. The disturbance estimation via three observers and real disturbance is shown in Figure 1, where the disturbance estimation of different observers can accurately estimate the real disturbance within 2 s. In addition, the designed AFXDO with faster transient performance can achieve accurate disturbance estimation in 0.5 s. In order to verify that AFXDO can make the disturbance estimation error converge within a fixed time, the disturbance estimation errors via different observers are shown in Figure 3. Noting that the convergence time of the disturbance estimation error with FIDO is affected by the initial disturbance estimation error, when the initial estimation is poor, the convergence time will be larger. Compared with the optimal control based on ESO and FIDO, the optimal control based on AFXDO achieves accurate disturbance estimation in 0.5 s. The settling time of disturbance estimation error is irrelevant to the initial perturbation estimation error, which is less than the theoretical settling time of 17/3 s from Theorem 1. In Figure 4, the observer gains of AFXDO are depicted. Due to the design of the observer without the upper bound of the disturbance, the observer gain is updated adaptively online by (10), avoiding the assumption of a known upper bound of the disturbance.

As shown in Figure 3 and Figure 4, AFXDO achieves the rapid convergence of disturbance estimation. Due to the rapid convergence of disturbance, which will further affect the tracking performance, the position tracking error and velocity tracking error are shown under three controllers in Figure 5 and Figure 6. The accuracy of the position tracking error under the three controllers reaches 10⁻¹ within 6 s. In addition, the trajectory tracking error of FXDO-ADP can enter steady-state performance within 5 s, which is smaller than that of the other two controllers, indicating that the rapid estimation of disturbance accelerates the convergence of the trajectory tracking error. As shown in Figure 5 and Figure 6, the velocity tracking errors with three controllers can enter the steady-state performance within 2 s. For ESO-ADP and FIDO-ADP, the tracking errors of FIDO-ADP can achieve steady-state performance within 1.3 s, which is faster than that of ESO-ADP. Further, the tracking errors of FXDO-ADP can enter the steady state within 1 s. Therefore, the transient performance of AFXDO-ADP is better than that of the ESO-ADP and FIDO-ADP controllers in terms of tracking errors of position and velocity. Due to the fact that AFXDO can make disturbance estimation follow real disturbance in a fixed time, the position and velocity tracking errors can enter a steady state within a short time.

Taking channel $x$ as an example, Figure 7 compares the control inputs of the three controllers, where the control inputs of FXDO-ADP and ESO-ADP are less. Combined with tracking errors in Figure 5 and Figure 6, the FXDO-ADP can achieve the rapid convergence of tracking errors with equivalent control consumption. Therefore, the controller makes the predefined cost function smaller among the three controllers.

• Case 2: Effect of weight update on learning performance

To explore the effectiveness of weight update for deriving the optimal policy, we compare the tracking performance and cost consumptions among the following schemes:

(1)

The designed ADP controller with a fixed learning rate (ADPFLR) [50]: The fixed learning rate is set as $\mathsf{\Upsilon }=100{\mathit{I}}_{15}$ to accelerate the convergence of tracking errors.

(2)

The designed ADP controller with a proposed variable learning rate (ADPVLR): The weight is updated by (30) rather than the fixed learning rate in ADPFLR. The parameters are selected as $\mathsf{\Upsilon }=100{\mathit{I}}_{15}$, $\delta =2.1$.

(3)

The actor–critic ADP controller with the gradient method [35] (ADP2): The learning rate in actor and critic NNs are set as ${\mathsf{\Upsilon }}_a=100{\mathit{I}}_{15}$, ${\Upsilon }_c=10I_{15}$.

Different from critic-only ADP, there exist the actor NN and critic NN to conduct the actor–critic ADP, which leads to the space and time complexity increasing due to the extra actor NN. In addition, on account of the variable learning rate designed with tracking errors, the space and time complexity are nonincreasing, where the tracking error is essential for other weight laws. The simulation results are shown in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13. In Figure 8 and Figure 9, the position tracking error and velocity tracking error under three controllers are shown, where different weight laws are employed under the ADP framework. The position tracking error enters the steady-state performance within 5 s, and the velocity tracking error converges to the neighborhood of origin within 2 s. In specific, the ADP based on critic NN has better transient performance than the ADP2 control based on the actor–critic NN in terms of velocity tracking errors and can achieve accurate tracking within 1 s, which is due to the hysteresis caused by weight convergence of the actor–critic NN.

Figure 10 and Figure 11 show the weight of the NN under the three controllers. The weights of the actor NN and critic NN in ADP2 are described in Figure 10, where the weight estimation of the critic NN and actor NN have obvious chattering at the initial stage. Conversely, the weight of the critic NN converges to an ideal weight based on critic-only ADP, as shown in Figure 11. Under the critic-only ADP framework, the weight update law of critic NN extracts the weight error through the auxiliary filtering operation, such that the weight update is driven by the weight error implied in the updated law, resulting in the fast convergence of weight. Furthermore, Figure 11 compares the weight of ADPFLR and ADPVLR. As shown in Figure 11, the ADPVLR significantly reduces the chattering occurring at the initial stage of ADPFLR with a fixed learning rate, which indicates that the proposed variable learning rate is superior to the weight update with a fixed learning rate. Specifically, its key features can be described thus: when the tracking error is large, the learning rate is reduced to avoid chattering; when the tracking error is small, the learning rate is increased to accelerate the convergence rate.

Since the convergence of weight affects the optimal control, the control inputs of three controllers in the $x$ channel are shown in Figure 12. Compared with ADPFLR and ADPVLR, the control input of ADPVLR is smaller, which can be due to the weight of the variable learning rate when avoiding the initial large control input. In addition, the control input of ADP2 is small, which leads to the slow convergence of tracking errors. Considering both tracking errors and control consumption, the cost functions of the three controllers are shown in Figure 13, where ADPFLR with a fixed learning rate requires high control consumption to achieve fast tracking performance. By the weight update law (30), the cost function of ADPVLR is decreased by 19.13% from that of ADPFLR. By introducing the weight updating law of variable learning rate, ADPVLR has a tradeoff ability in control consumption and tracking performance, which not only achieves better tracking performance but also ensures lower control consumption.

By analyzing the effectiveness of disturbance observer and new weight law, although AFXDO is carried with large control consumption to achieve faster convergence, by combining AFXDO and weight law with a variable learning rate, the optimal control consumption is reduced, so the total control consumption is decreased, which induces lower cost functions.

Different from the uncertainty of the above system, $\Delta =\Delta \mathit{f}(\mathit{x})+\mathit{d}(t)$, we set the only $\mathit{d}(t)$ to unknown for a sensitive analysis of the system. The tracking performance of position and velocity are shown in Figure 14 and Figure 15. The position and velocity can follow the reference position and velocity in 4 s. The weight of the critic NN is described in Figure 16, avoiding the initial chattering phenomenon. We will validate the effectiveness of the completely unknown system in actual application.

6. Conclusions

The approximate tracking control with adaptive fixed-time estimation is developed to address the tracking problem of partially unknown nonlinear systems. After the procession of feedforward control for the error system, the novel adaptive fixed-time disturbance observer is designed such that the estimation error for disturbance can converge to zero within a fixed time. Then, an optimal control policy is derived by the critic-only ADP framework, where a novel weight adaptive law with a variable learning rate is proposed for updating weight without initial oscillation phenomena. The tracking errors and weight estimation errors can be proven to be UUB. The designed approximate optimal policy approaches the optimal control. The superiority of the proposed controller can be verified in the simulation, which well balances tracking performance and control consumption. Noting that control input and tracking error may exceed the capacity of the actuator and predefined region, which is not considered in controller design, future work will concentrate on the extension of the proposed controller with a new cost function for a partially unknown nonlinear system with input saturation and output constraints to reduce control consumption and improve tracking performance.