Reinforcement Learning-Based Adaptive Position Control Scheme for Uncertain Robotic Manipulators with Constrained Angular Position and Angular Velocity

Abstract

Aiming at robotic manipulators subject to system uncertainty and external disturbance, this paper presents a novel adaptive control scheme that uses the time delay estimation (TED) technique and reinforcement learning (RL) technique to achieve a good tracking performance for each joint of a manipulator. Compared to conventional controllers, the proposed control scheme can not only handle the system parametric uncertainty and external disturbance but also guarantee both the angular positions and angular velocities of each joint without exceeding their preset constraints. Moreover, it has been proved by using Lyapunov theory that the tracking errors are uniformly ultimately bounded (UUB) with a small bound related to the parameters of the controller. Additionally, an innovative RL-based auxiliary term in the proposed controller further minimizes the steady state tracking errors, and thereby the tracking accuracy is not compromised by the lack of asymptotic convergence of tracking errors. Finally, the simulation results validate the effectiveness of the proposed control scheme.

1. Introduction

Robotic manipulators have been extensively applied in assisting or even replacing humans to perform the various tasks such as assembling [1], machine operation [2], deburring [3], drilling [4], transportation [5] and manufacturing [6,7]. In order to successfully perform such industrial tasks, it is essential to well control the motion of robotic manipulators. However, the highly nonlinear and uncertain dynamics of robotic manipulators result in the inapplicability of many traditional controllers designed for linear systems such as linear quadratic control (LQC) [8,9] and linear H∞ control [10,11]. Hence, many researchers have been motivated to place efforts on designing advanced controllers for robotic manipulators.

State feedback linearization requiring a known dynamic model can transform the nonlinear system to a linear form [12]. However, the dynamic model of robotic manipulators is always uncertain because of unknown external disturbances and uncertain system parameters. To handle the issue of uncertainty, many efforts have been made such as sliding mode control (SMC) [13,14,15,16,17,18], fuzzy logic system (FLS)-based control [19,20,21], neural network (NN)-based control [22,23,24,25,26], and disturbance observer (DOB)-based control [27,28,29]. More precisely, Zhang et al. [13] propose a fixed time sliding mode control for uncertain robotic manipulators in which a conservative switching gain requiring the upper bound of lumped uncertainty is used. In [21], an adaptive controller-based T-S (Takagi-Sugeno) fuzzy system is designed, and the modified T-S fuzzy system can efficiently approximate the unknown model of robotic manipulators. Hu et al. [26] present a multiple-layer neural network-based controller that can achieve a high accuracy of motion control of robotic manipulators subject to unknown disturbances. In [28], a high-order-sliding-mode differentiator (HOSMD)-based estimator is designed to compensate the mismatched uncertainties without the need of assuming a bounded uncertainty.

Besides the uncertainty of the dynamic model, state constraint is also a common problem of robotic manipulators, which has attracted much attention. For example, Sun et al. [30] proposed an adaptive neural network control scheme considering full-state constraints of robotic manipulators. In [31], an adaptive fuzzy control scheme that can guarantee the constrained output is presented. Yu et al. [32] designed an adaptive fuzzy control scheme working with a disturbance observer for the manipulators with full-state constraints. Nevertheless, those controllers can only achieve either the constrained tracking error of joint angle or the constrained error between the actual angular velocity and the virtual control signal. In other words, controllers in [30,31,32] do not guarantee the angular position and angular velocity for each joint of the robotic manipulator, within the preset constraints. Recently, Yang et al. [33] designed a new adaptive control scheme that can guarantee the angular position for each joint of a robotic manipulator to never exceed the preset constraints. However, the constrained angular velocity of each joint is not guaranteed.

In the light of reviewing the existing literature, the following issues need to be further solved:

• In order to safely perform robotic manipulators, both the angular position and angular velocity of each joint of robotic manipulators should be controlled to not exceed the preset constraints. More precisely, the angular position (rotation angle) of each joint should be always within a reasonable range to have no risk on physically breaking the joint. Similarly, the angular velocity of each joint should not exceed its maximum related to the maximum rotational speed of the driving motor;
• For some existing controllers (e.g., [9,23,26,30,31,32]), the tracking accuracy could be compromised due to the bounded result of tracking errors and uncertainty-estimation errors. Therefore, it is needed to avoid the loss of tracking accuracy caused by the lack of asymptotic convergence of tracking errors.

To handle the above issues, this paper proposes a new adaptive control scheme that utilizes time delay estimation (TDE) and reinforcement learning (RL) for $n$-link robotic manipulators. In the proposed control scheme, the multiple-input-and-multiple-output (MIMO) robotic system is initially decomposed into $n$ single-input-single-output (SISO) subsystems by TDE. Each subsystem is with an unknown bounded TDE error. After that, the novel virtual control law for each subsystem is designed, which can not only achieve the boundness of tracking errors for each joint in the presence of TDE error, but can also guarantee both the angular position and angular velocity for each joint to be not exceeding the preset constraints. To improve the tracking accuracy, an RL-based term in the virtual control law is designed, which automatically learns the optimal parameters of the controller in the different system states. RL is an artificial intelligence technique that gradually explores the optimal policy by interacting with the environment, which has attracted many interests in the control of robotic manipulators such as [34,35,36]. Particularly, the RL-based term in this paper is designed to avoid the violation of the boundness of tracking errors, even if a bad policy is tried by RL, which ensures a safe environment for implementing RL. The tracking errors are proven to be uniformly ultimately bounded (UUB) via Lyapunov theory. Simulation results indicate the proposed control scheme can achieve a high tracking accuracy in the presence of model uncertainty and unknown disturbances.

The major merits of this paper include the following points:

• Compare with some existing research [30,31,32,33], in addition to the basic achievement of the uniformly ultimately bounded (UUB) tracking error for each joint in the presence of TDE error; the control scheme can guarantee both the angular position and angular velocity for each joint to be not exceeding the preset constraints;
• The novel adaptive gain in (13) results in the smooth control torques to reduce the chattering effect caused by switching term in (10). Meanwhile, an RL-based term can effectively improve the tracking accuracy, which thereby reduces the possible steady-state tracking errors caused by the lack of asymptotic convergence;
• The mathematical expression of the controller is simple, meanwhile, any prior knowledge of upper bounds caused by an imprecise model are unnecessary in our control scheme.

The rest of this paper is organized as follows: in Section 2, the dynamics model of n-link robotic manipulators is given, and the control objective is described. In Section 3, the RL-based adaptive control scheme is proposed and the proof of stability is given. In Section 4, the numerical simulation is conducted to verify the effectiveness of the proposed controller. The conclusion is given in Section 5.

2. Dynamical Model and Problem Statement

The dynamic model of n-link robot manipulators is shown as the following:

$$\mathit{M}\left(q\left(t\right)\right)\ddot{\mathit{q}}\left(t\right)+\mathit{C}\left(\mathit{q}\left(t\right),\dot{\mathit{q}}\left(t\right)\right)\dot{\mathit{q}}\left(t\right)+\mathit{G}\left(\mathit{q}\left(t\right)\right)+\mathit{F}\left(\dot{\mathit{q}}\left(t\right)\right)=\mathit{\tau }\left(t\right)+{\mathit{\tau }}_{\mathit{d}}\left(\mathit{t}\right)$$

(1)

where $\mathit{M}\left(\mathit{q}\left(t\right)\right)$ ∈ R^n×n is the inertia matrix, $\mathit{q}\left(t\right)={\left[q_1\left(t\right),q_2\left(t\right),..,q_i\left(t\right),..,q_n\left(t\right)\right]}^T$∈ Rⁿ is the vector of angular positions of joints of manipulator. $\mathit{C}\left(\mathit{q}\left(t\right),\dot{\mathit{q}}\left(t\right)\right)$ ∈ R^n×n is the Coriolis and centrifugal matrix. $\mathit{G}\left(\mathit{q}\left(t\right)\right)$∈ Rⁿ is the gravity vector. $\mathit{F}\left(\dot{\mathit{q}}\left(t\right)\right)$∈ Rⁿ is the vector of friction. $\mathit{\tau }$(t) ∈ Rⁿ is the vector of torques applied on the joints. ${\mathit{\tau }}_{\mathit{d}}\left(\mathit{t}\right)\in R^n$ is the external disturbance.

The model (1) can be further written as (2) to indicate the system uncertainty.

$$\begin{gathered} \ddot{\mathit{q}}\left(\mathit{t}\right)=\mathit{M}{\left(\mathit{q}\left(t\right)\right)}^{-1}\left[-\mathit{C}\left(\mathit{q}\left(t\right),\dot{\mathit{q}}\left(t\right)\right)\dot{\mathit{q}}\left(t\right)-\mathit{G}\left(\mathit{q}\left(t\right)\right)-\mathit{F}\left(\dot{\mathit{q}}\left(t\right)\right)+{\mathit{\tau }}_{\mathit{d}}\left(\mathit{t}\right)\right]+\left[\mathit{M}{\left(\mathit{q}\left(t\right)\right)}^{-1}-\widehat{\mathit{M}}{\left(\mathit{q}\left(t\right)\right)}^{-1}\right]\mathit{\tau }\left(t\right) \\ +\widehat{\mathit{M}}{\left(\mathit{q}\left(t\right)\right)}^{-1}\mathit{\tau }\left(t\right)=\mathit{\Gamma }\left(t\right)+\widehat{\mathit{M}}{\left(\mathit{q}\left(t\right)\right)}^{-1}\mathit{\tau }\left(t\right) \end{gathered}$$

(2)

where $\widehat{\mathit{M}}\left(\mathit{q}\left(t\right)\right)$ is the estimation of $\mathit{M}\left(q\left(t\right)\right)$. $\mathit{\Gamma }\left(\mathit{t}\right)=\mathit{M}{\left(\mathit{q}\left(t\right)\right)}^{-1}\left[-\mathit{C}\left(\mathit{q}\left(t\right),\dot{\mathit{q}}\left(t\right)\right)\dot{\mathit{q}}\left(t\right)-\mathit{G}\left(\mathit{q}\left(t\right)\right)-\mathit{F}\left(\dot{\mathit{q}}\left(t\right)\right)+{\mathit{\tau }}_{\mathit{d}}\left(\mathit{t}\right)\right]+\left[\mathit{M}{\left(\mathit{q}\left(t\right)\right)}^{-1}-\widehat{\mathit{M}}{\left(\mathit{q}\left(t\right)\right)}^{-1}\right]\mathit{\tau }\left(t\right)$ is the system uncertainty.

The vector of error between the desired angular position and actual angular position is defined in (3):

$$\mathit{e}\left(t\right)=\mathit{q}\left(t\right)-{\mathit{q}}_d$$

(3)

where ${\mathit{q}}_d={[q_{d1},q_{d2},\dots ,q_{dn}]}^T\in R^n$ is the vector of desired angular position of joints. $\mathit{e}={[e_1,e_2,\dots ,e_n]}^T\in R^n$ is the tracking error vector.

The main control objective is to design the torque $\mathit{\tau }\left(\mathit{t}\right)$ that can drive all the joints of the robotic manipulator system (1) to approach their desired angular positions. Meanwhile, both the angular positions and angular velocities for each joint should be guaranteed to not exceed the given constraints. The control objective can be described by (4)–(6).

$$\left|\left|e\left(t\right)\right|\right|\le \sigma , \forall t\ge 0$$

(4)

$$\left|q_i\left(t\right)\right|<{\epsilon }_i, \forall t\ge 0, i=1,2,..,n$$

(5)

$$\left|{\dot{q}}_i\left(t\right)\right|<{\Lambda }_i, \forall t\ge 0, i=1,2,..,n$$

(6)

where ${\sigma }^{*}>\sigma \ge 0$, and ${\sigma }^{*}>0$ is a positive constant, ${\epsilon }_i>0$ is a positive constant referring to the angular position constraint of the $i^{th}$ joint, and ${\Lambda }_i>0$ is a positive constant, meaning the angular velocity constraint of the $i^{th}$ joint.

Remark 1. 

The positive constant ${\sigma }^{*}$ is related to the initial state of the system and parameters of the controller. $\sigma$ reflects the tracking accuracy. Notably, $\sigma \left(t\to \infty \right)=0$ means the asymptotic convergence of tracking errors.

Remark 2. 

In this paper, we consider the angular position tracking problem for each joint of the manipulator. Hence, $q_{di}$ is a constant for $i=1,2,..,n.$, which means ${\dot{q}}_{di}=0$. The angular trajectory tracking problem will be considered in our future work.

3. Controller Design and Stability Analysis

In this part, the adaptive RL-based controller working with TDE is developed. After that, the stability is proven by using Lyapunov theory.

3.1. Controller Design

The TDE technique is applied to handle the system uncertainty in (2):

$$\widehat{\mathit{\Gamma }}\left(t\right)\approx \mathit{\Gamma }\left(t-L\right)=\ddot{\mathit{q}}\left(t-L\right)-{\widehat{\mathit{M}}}^{-\mathbf{1}}\left(\mathit{q}\left(\mathit{t}-\mathit{L}\right)\right)\mathit{\tau }\left(t-L\right)$$

(7)

where $\widehat{\mathit{\Gamma }}\left(t\right)={[{\widehat{\Gamma }}_1,{\widehat{\Gamma }}_2,..,{\widehat{\Gamma }}_n]}^T\in R^n$ is the estimate of $\Gamma \left(t\right)$. $L>0$ is the sampling time of TDE.

Lemma 1. 

[14] The TDE error of robotic manipulator (2) is bounded such that $\left|{\Gamma }_i\left(t\right)-{\widehat{\Gamma }}_i\left(t\right)\right|\le {\Gamma }_i^{*}$ (for $i=1,2,\dots ,n$) if the following condition is satisfied:

$$\left|\right|\mathit{I}-{\mathit{M}}^{-1}\left(\mathit{q}\left(t\right)\right)\widehat{\mathit{M}}{\left(\mathit{q}\left(t-L\right)\right)\left|\right|}_{\mathbf{2}}<1$$

(8)

where ${\Gamma }_i^{*}$ is an unknown positive constant.

The control law working with TDE technique is designed as follows:

$$\mathit{\tau }\left(t\right)={\widehat{\mathit{M}}}^{}\left(\mathit{q}\left(t\right)\right)\left[-\widehat{\mathit{\Gamma }}\left(t\right)+\mathit{u}\left(t\right)\right]$$

(9)

where $\mathit{u}\left(t\right)={[u_1\left(t\right),u_2\left(t\right),\dots ,u_n\left(t\right)]}^T\in R^n$ is the virtual control law.

The virtual control law in (9) is designed as follows:

$$\begin{gathered} u_i=-{\widehat{d}}_{i}sgn\left({\dot{q}}_i\right)-\frac{1}{1+k_{yi}{C}_i}\left[k_{pi}{e}_i+k_{di}{\dot{q}}_i+k_{si}{A}_i{q}_i+k_{si}{B}_i{z}_i+{\lambda }_i {\dot{q}}_i+{\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\right] \\ i=1, 2, \dots ,n \end{gathered}$$

(10)

where $k_{pi}$, $k_{si}$, $k_{di}, {₵}_i$ and $k_{yi}$ are the positive constants determined by users. ${\lambda }_i>0$ is a positive variable determined by the fuzzy reinforcement learning mechanism.

$$A_i=\frac{{\epsilon }_i^2{z}_i^2}{{\left({\epsilon }_i^2-q_i^2\right)}^2}, B_i=\frac{{\epsilon }_i^2}{{\epsilon }_i^2-q_i^2}, C_i=\frac{2{\Lambda }_i^2}{{\left({\Lambda }_i^2-{\dot{q}}_i^2\right)}^2}$$

(11)

where ${\epsilon }_i>0$ is the restricted upper bound of angular position of the $i^{th}$ joint. ${\mathsf{\Lambda }}_i>0$ is the restricted upper bound of angular velocity of the $i^{th}$ joint.

And the variable $z_i$ is defined as follows:

$$z_i=e_i+{\int }_0^t{\eta }_i\left(\mathsf{\Theta }\right)d\mathsf{\Theta }, {\eta }_i=-{\beta }_i{z}_i$$

(12)

where ${\beta }_i$ is a positive constant.

The ${\widehat{d}}_i$ is used to handle the bounded TDE error and the update law of ${\widehat{d}}_i$ is designed as follows.

$${\dot{\widehat{d}}}_i=\{\begin{array}{c}{\psi }_i\left(1+k_{yi}{C}_i\right)\left|{\dot{q}}_i\right| , if\left({\widehat{d}}_i\le 0\right) or \left({\Omega }_i>{\overline{\Omega }}_i\right)\\ \\ -{\psi }_i\frac{{\delta }_i}{\left(1+k_{yi}{C}_i\right)\left|{\dot{q}}_i\right|} , if\left({\widehat{d}}_i>0\right) and \left({\Omega }_i\le {\overline{\Omega }}_i\right)\end{array}$$

(13)

where ${\psi }_i$, ${\delta }_i$ and ${\overline{\Omega }}_i$ are positive constants. The variable ${\Omega }_i$ is defined as (14).

$${\Omega }_i=\frac{1}{2}{\dot{q}}_i^2+\frac{1}{2}{k}_{pi}{e}_i^2+\frac{1}{2}{k}_{si}\frac{{\epsilon }_i^2{z}_i^2}{{\epsilon }_i^2-q_i^2}+k_{yi}\frac{{\dot{q}}_i^2}{{\Lambda }_i^2-{\dot{q}}_i^2}+{\lambda }_i{₵}_i\ln \left[\cosh \left(\frac{e_i }{{₵}_i}\right)\right]$$

(14)

Remark 3. 

${\widehat{d}}_i$ is to guarantee the stability in the presence of the bounded TDE errors. A conservative update law of ${\widehat{d}}_i$ is to monotonously increase the value of ${\widehat{d}}_i$ such that ${\dot{\widehat{d}}}_i={\psi }_i\left(1+k_{yi}{C}_i\right)\left|{\dot{q}}_i\right|$. Although such adaptive law can achieve the bounded tracking errors and angular velocities, the great value of ${\widehat{d}}_i$ could result in the chattering effect on the calculated control torques. Therefore, a novel adaptive law for ${\widehat{d}}_i$ shown in (13) is proposed to mitigate the chattering effect by decreasing the value of ${\widehat{d}}_i$ without the breach of stability of system. The proof will be given later. Moreover, ${\widehat{d}}_i>0$ holds because ${\widehat{d}}_i\le 0$ leads to ${\dot{\widehat{d}}}_i={\psi }_i\left(1+k_{yi}{C}_i\right)\left|{\dot{q}}_i\right|\ge 0$.

Remark 4. 

Similar to [33], the terms $A_i{q}_i$ and $B_i{z}_i$ in (10) are to guarantee the angular position of each joint of manipulator to not exceed the preset constraint $\pm {\epsilon }_i$. While the term $\frac{1}{1+k_{yi}{C}_i}$ in (10) is to guarantee the angular velocity of each joint to not exceed the preset constraint $\pm {\Lambda }_i$, which was not achieved in [33]. The proof will be detailed later.

3.2. Stability Analysis

Theorem 1. 

If the initial angular position and velocity of all joints are within their preset constraints such that $\left|q_i\left(0\right)\right|<{\epsilon }_i$ and $\left|{\dot{q}}_i\left(0\right)\right|<{\Lambda }_i$, and (8) in lemma 1 holds, the control law consisting of (7) and (9)–(14) can achieve the uniformly ultimately bounded tracking errors of robotic manipulator system (1). Meanwhile, the angular velocity and angular position of each joint of manipulator are within the preset constraints such that $\left|q_i\left(t\right)\right|<{\epsilon }_i$ and $\left|{\dot{q}}_i\left(t\right)\right|<{\Lambda }_i$, $\forall t>0$. Namely, the control target (4)–(6) is achieved.

Proof. 

By inserting (9) into (2) and using the fact of ${\widehat{M}}^{-1}\widehat{M}=I$ with $I=diag\left(\left[1,1,..,1\right]\right)\in R^{n\times n}$, the MIMO robotic manipulator system can be decoupled into $n$ uncertain SISO subsystems (15).

$${\ddot{q}}_i\left(t\right)=u_i\left(t\right)+d_i\left(t\right)i=1,2,\dots ,n$$

(15)

where $q_i\in q$ is the angular position of the $i^{th}$ joint. $u_i$ is designed in (10). $d_i\left(t\right)={\Gamma }_i\left(t\right)-{\widehat{\Gamma }}_i\left(t\right)$ is the TDE error.

The following Lyapunov candidate is designed:

$$V={\displaystyle \sum }_{i=1}^n{V}_i={\displaystyle \sum }_{i=1}^n[\frac{1}{2}{\dot{q}}_i^2+\frac{1}{2}{k}_{pi}{e}_i^2+\frac{1}{2{\psi }_i}{\tilde{d}}_i^2+k_{si}\frac{{\epsilon }_i^2{z}_i^2}{2({\epsilon }_i^2-q_i^2)}+k_{yi}\frac{{\dot{q}}_i^2}{{\Lambda }_i^2-{\dot{q}}_i^2}+{\lambda }_i {\Lambda }_i{₵}_i\ln (\cos \mathrm{h}(\frac{e_i}{{₵}_i})) ]$$

(16)

where ${\tilde{d}}_i={\Gamma }_i^{*}-{\widehat{d}}_i$. ${\Gamma }_i^{*}$ is the upper bound TDE error defined in Lemma 1.

Remark 5. 

Clearly, $k_{si}\frac{{\epsilon }_i^2{z}_i^2}{2({\epsilon }_i^2-q_i^2)}>0$ holds as long as $\left|q_i\right|<{\epsilon }_i$. $k_{yi}\frac{{\dot{q}}_i^2}{{\Lambda }_i^2-{\dot{q}}_i^2}>0$ as long as $\left|{\dot{q}}_i\right|<{\Lambda }_i$. Furthermore, ${\Lambda }_i{₵}_{i}ln\left(cosh\left(\frac{e_i}{{₵}_i}\right)\right) \ge 0$ holds because of the fact of $cosh(·)\ge 1$ and the fact of $ln\left(x\right)\ge 0$ with $x\ge 1$. Therefore, the Lyapunov candidate (16) is positive defined as long as $\left|q_i\right|<{\epsilon }_i$ and $\left|{\dot{q}}_i\right|<{\Lambda }_i$.

Taking the derivative of the Lyapunov function (16) with respect to the time t and using (11), (12) and (15), we have:

$$\begin{array}{l}\dot{V}={\displaystyle \sum }_{i=1}^n[{\dot{q}}_i{\ddot{q}}_i+k_{pi}{e}_i{\dot{q}}_i-\frac{1}{{\psi }_i}{\tilde{d}}_i{\dot{\widehat{d}}}_i+ k_{si}{A}_i{q}_i{\dot{q}}_i+k_{si}{B}_i{z}_i{\dot{z}}_i+k_{yi}{C}_i{\dot{q}}_i{\ddot{q}}_i+{\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right){\dot{q}}_i] \\ ={\displaystyle {\displaystyle \sum }_{i=1}^n}{\dot{q}}_i\left[\left(1+k_{yi}{C}_i\right)\left(u_i+d_i\right)+k_{pi}{e}_i+ k_{si}{A}_i{q}_i+ k_{si}{B}_i{z}_i +{\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\right]\\ - {\displaystyle \sum }_{i=1}^n\left(\frac{1}{{\psi }_i}{\tilde{d}}_i{\dot{\widehat{d}}}_i+ k_{si}{B}_i{z}_i {\eta }_i\right)\end{array}$$

(17)

Substituting (10) and into (17) with the fact of $C_i>0$ as long as $\left|{\dot{q}}_i\right|<{\Lambda }_i$:

$$\begin{gathered} \dot{V}={\displaystyle \sum }_{i=1}^n{\dot{q}}_i\left(1+k_{yi}{C}_i\right)\left[-{\widehat{d}}_{i}sgn\left({\dot{q}}_i\right)+d_i\right]-{{\displaystyle \sum }}_{i=1}^n\frac{1}{{\psi }_i}{\tilde{d}}_i{\dot{\widehat{d}}}_i \\ -{{\displaystyle \sum }}_{i=1}^n k_{si}{B}_i{\beta }_i{z}_i^2-{{\displaystyle \sum }}_{i=1}^n\left({\lambda }_i+k_{di}\right){\dot{q}}_i^2 \\ \le {\displaystyle \sum }_{i=1}^n\left|{\dot{q}}_i\right|\left(1+k_{yi}{C}_i\right)\left({\Gamma }_i^{*}-{\widehat{d}}_i\right)-{{\displaystyle \sum }}_{i=1}^n\frac{1}{{\psi }_i}{\tilde{d}}_i{\dot{\widehat{d}}}_i \\ ={\displaystyle \sum }_{i=1}^n\left[\left|{\dot{q}}_i\right|\left(1+k_{yi}{C}_i\right)-\frac{1}{{\psi }_i}{\dot{\widehat{d}}}_i\right]\left({\Gamma }_i^{*}-{\widehat{d}}_i\right) \end{gathered}$$

(18)

Therefore, for each $V_i$, we can derive (19) by combining (13) and (18):

$${\dot{V}}_i\le \{ \begin{array}{c}0 , if\left({\widehat{d}}_i\le 0\right) or \left({\Omega }_i\ge {\overline{\Omega }}_i\right) \\ \\ {\mathfrak{x}}_i\left({\Gamma }_i^{*}-{\widehat{d}}_i\right), if\left({\widehat{d}}_i>0\right) and \left({\Omega }_i<{\overline{\Omega }}_i\right)\end{array}$$

(19)

where ${\mathfrak{x}}_i=\left|{\dot{q}}_i\right|\left(1+k_{yi}{C}_i\right)+\frac{{\delta }_i}{\left|{\dot{q}}_i\right|\left(1+k_{yi}{C}_i\right)}$. Clearly, ${\mathfrak{x}}_i\ge 2\sqrt{{\delta }_i}>0$ holds.

In addition, the $V_i$ defined in (16) can be written as:

$$V_i={\Omega }_i+\frac{1}{2{\psi }_i}{\tilde{d}}_i^2$$

(20)

where ${\Omega }_i$ is defined in (14).

To prove the Lyapunov function, $V_i$ is bounded by a positive constant and we assume a sufficiently large constant $V_i^{*}$. Clearly, the sufficiently large $V_i^{*}$ requires at least one of the terms (${\Omega }_i$ or ${\tilde{d}}_i^2$) to be sufficiently large. If ${\Omega }_i$ is sufficiently large such that ${\Omega }_i>{\overline{\Omega }}_i$, then ${\dot{V}}_i\le 0$ holds according to (19). If ${\tilde{d}}_i^2={({\Gamma }_i^{*}-{\widehat{d}}_i)}^2$ is sufficiently large such that ${\tilde{d}}_i^2>{(2{\Gamma }_i^{*})}^2$, then ${\widehat{d}}_i>3{\Gamma }_i^{*}$ will hold because of the facts of ${\Gamma }_i^{*}>0$ (Lemma 1) and ${\widehat{d}}_i\ge 0$ (remark 3), which further means ${\Gamma }_i^{*}-{\widehat{d}}_i<0$ holds. As a result, according to (19) and the fact of ${\mathfrak{x}}_i\ge 2\sqrt{{\delta }_i}>0$, ${\dot{V}}_i\le 0$ can hold by ${\Omega }_i>{\overline{\Omega }}_i$ or ${\tilde{d}}_i^2>{(2{\Gamma }_i^{*})}^2$, which means ${\dot{V}}_i\le 0$ holds if $V_i\ge {\overline{\Omega }}_i+\frac{1}{2{\psi }_i}{(2{\Gamma }_i^{*})}^2$ holds. It is thereby easy to conclude that $V_i$ is bounded by $V_i^{*}$ such that $V_i\le V_i^{*}=\max \{V_i\left(0\right), {\overline{\Omega }}_i+\frac{1}{2{\psi }_i}{(2{\Gamma }_i^{*})}^2\}$.

Therefore, the tracking error of the $i^{th}$ subsystem is bounded because of the fact of $\frac{1}{2}{k}_{pi}{e}_i^2\le V_i$.

$$|e_i|\le \sqrt{\frac{2}{k_{pi}}\max \{V_i\left(0\right), ({\overline{\Omega }}_i+\frac{1}{2{\psi }_i}\left(2{\Gamma }_i^{*}{)}^2\right)}i=1,2,..,n$$

(21)

Then, the norm of the vector of the tracking error is thereby bounded.

$$\left|\left|e\left(t\right)\right|\right|\le \sqrt{{\displaystyle \sum }_{i=1}^n\frac{2}{k_{pi}}\max \{V_i\left(0\right),[{\overline{\Omega }}_i+\frac{1}{2{\psi }_i}\left(2{\Gamma }_i^{*}{)}^2\right]}$$

(22)

Moreover, the bounded $V_i$ implies all terms in $V_i$ are bounded. The (23) therefore holds, which means $\left|q_i\right|<{\epsilon }_i$ and $\left|{\dot{q}}_i\right|<{\mathsf{\Lambda }}_i$ hold.

$$\frac{{\epsilon }_i^2{z}_i^2}{2({\epsilon }_i^2-q_i^2)}, \frac{{\dot{q}}_i^2}{{\Lambda }_i^2-{\dot{q}}_i^2}\in L_{\infty }i=1,2,\dots ,n$$

(23)

□

3.3. Fuzzy Q Reinforcement Learning Mechanism Determining Parameters of Controller

In this section, a fuzzy Q reinforcement learning mechanism is designed to tune the parameter ${\lambda }_i$ to improve the tracking accuracy. The motivations of RL are detailed in remark 7.

Lemma 2. 

If (24) holds, the tracking error $e_i$ defined in (3) will asymptotically converge to zero with the converging rate satisfying $\left|{\dot{q}}_i\right|<{\Lambda }_i$.

$$0={\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right), i=1,2,..n$$

(24)

Proof. 

A simple Lyapunov function is given in (25).

$${\mathcal{V}}_i=\frac{1}{2}{e}_i^2$$

(25)

Combining (24) and derivative of (25), we can obtain (26).

$${\dot{\mathcal{V}}}_i=-{\Lambda }_i{e}_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\le -{\Lambda }_i{₵}_{i}tan{h}^2\left(\frac{e_i }{{₵}_i}\right)\le 0$$

(26)

${\dot{\mathcal{V}}}_i\le 0$ implies ${\mathcal{V}}_i\in L_{\infty }$. And then, (27) can be obtained by integration on both sides of (26).

$${\mathcal{V}}_i(\infty )-{\mathcal{V}}_i\left(0\right)\le -{\Lambda }_i{₵}_i{\int }_0^{\infty }tan{h}^2\left(\frac{e_i }{{₵}_i}\right)$$

(27)

In the light of Barbalet’s Lemma, (27) implies $tanh\left(\frac{e_i(\infty ) }{{₵}_i}\right)=0$, which means $e_i(\infty )=0$. Moreover, it is clear that $\left|{\dot{q}}_i|=|{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\right|<{\mathsf{\Lambda }}_i$ due to the fact of $\tanh (·)<1$.

□

Remark 6. 

The asymptotic convergence of tracking errors in Lemma 2 is stronger than the boundness of tracking errors that we achieved in Theorem 1, which means a better tracking accuracy. It is because asymptotic convergence means $e_i$ is eventually going to zero, while boundedness only implies $\left|e_i\right|$ bounded by a positive constant. Therefore, the proposed controller can be improved with a better tracking accuracy by finding the optimal parameters of the proposed controller that are able to minimize $\left|{\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\right|$.

The converging behaviour of tracking errors of (24) is visualized in Figure 1.

Lemma 3. 

For a moment $t^{*}$, if the system states $\left(q_i\left(t^{*}\right),{\dot{q}}_i\left(t^{*}\right)\right)$ satisfy (5) and (6) as well as ${\dot{q}}_i\left(t^{*}\right)+{\Lambda }_{i}tanh\left(\frac{e_i\left(t^{*}\right) }{{₵}_i}\right)\ne 0$ holds, there always exists the optimal parameter ${\lambda }_i^{*}>0$ to decrease ${({\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right))}^2$. Namely, (28) holds.

$$\frac{d[\left({\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right){)}^2\right]}{dt}{|}_{t=t^{*}}<0$$

(28)

Proof. 

Combining (10) and (15), we can obtain the 1st derivative of ${\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)$ with respect to time, shown as (29).

$$\begin{gathered} \frac{d({\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right))}{dt}=-{\widehat{d}}_{i}sgn\left({\dot{q}}_i\right)+d_i\left(t\right)+\frac{{\Lambda }_i }{{₵}_i}\frac{\partial tanh\left(\frac{e_i }{{₵}_i}\right)}{\partial \frac{e_i }{{₵}_i}}{\dot{q}}_i-\frac{1}{1+k_{yi}{C}_i}\left[k_{pi}{e}_i+k_{di}{\dot{q}}_i+k_{si}{A}_i{q}_i+ \\ {k}_{si}{B}_i{z}_i\right]-\frac{1}{1+k_{yi}{C}_i}[{\lambda }_i {\dot{q}}_i+{\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)] \end{gathered}$$

(29)

Then, we define a negative changing rate of ${({\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right))}^2$ at moment $t^{*}$ in (30).

$$\frac{d}{dt}{[{\dot{q}}_i\left(t\right)+{\Lambda }_{i}tanh\left(\frac{e_i\left(t\right)}{{₵}_i}\right)]}^2{|}_{t=t^{*}}<0$$

(30)

Using (5), (6), (29) and (30), a solution of ${\lambda }_i^{*}$ satisfying (30) is obtained in (31).

$${\lambda }_i^{*}>\frac{1}{|{\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)|}[\left(1+k_{yi}{C}_i\right)({\widehat{d}}_i+{\Gamma }_i^{*}+\frac{{\Lambda }_i^2 }{{₵}_i})+k_{pi}\left|e_i\right|+k_{di}{\Lambda }_i+k_{si}\left|A_i\right|{\epsilon }_i+k_{si}\left|B_i\right|\left|z_i\right|]$$

(31)

Notably, ${\widehat{d}}_i$, $\left|e_i\right|$, $\left|z_i\right|$, $\left|A_i\right|$ and $\left|B_i\right|$ are all bounded because of the bounded Lyapunov function (16). Therefore, a finite real solution of ${\lambda }_i^{*}$ satisfying (31) exists as long as ${\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)\ne 0$.

□

Remark 7. 

According to Lemma 3 and Remark 6, the optimal parameters ${\lambda }_i^{*}$ leading to a decrease of $|{\dot{q}}_i+{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)|$ at the moment $t^{*}$ can improve the tracking accuracy. Although a large enough ${\lambda }_i$ can be selected to satisfy (31), an inappropriately large ${\lambda }_i$ could lead to a significant chattering of control torques and thereby compromise the tracking accuracy. Moreover, an optimal ${\lambda }_i$ is hard to be deterministically found due to the complexity of the system and the unknown TDE errors. Therefore, a fuzzy Q RL mechanism is designed to automatically determine the optimal ${\lambda }_i^{*}$.

Fuzzy Q learning is a common version of RL applicable on continuous systems, which can explore the optimal policy by interacting with the environment [37]. In this paper, the ${\lambda }_i$ is to be tuned by a fuzzy Q learning mechanism according to the tracking error $e_i$ and the angular velocity ${\dot{q}}_i$. The linguistic rules to determine ${\lambda }_i$ can be given as the following form:

$$\mathrm{IF} e_i\left(\mathbb{k}\right) \mathrm{is} {ℒ}_{e,i} \mathrm{AND} {\dot{q}}_i\left(\mathbb{k}\right) \mathrm{is} {ℒ}_{\dot{q,i}}, \mathrm{THEN} {\lambda }_i\left(\mathbb{k}\right) \mathrm{is} {ℒ}_{\lambda ,i}$$

(32)

${ℒ}_{e,i}$, ${ℒ}_{\dot{q,i}}$ and ${ℒ}_{\lambda ,i}$ are the linguistic descriptions of tracking error $e_i$, angular velocity ${\dot{q}}_i$ and parameter ${\lambda }_i$ respectively. $\mathbb{k}$ is the current moment. The linguistic description could be “small”, “medium” and “big”. As a result, an example of linguistic rule could be: IF $e_i\left(\mathbb{k}\right)$ is $small$ AND ${\dot{q}}_i\left(\mathbb{k}\right)$ is $small$, THEN ${\lambda }_i\left(\mathbb{k}\right)$ is $small$.

Some intuitive linguistic rules can be given as:

• IF $e_i\left(\mathbb{k}\right)$ is small, and ${\dot{q}}_i\left(\mathbb{k}\right)$ is small THEN ${\lambda }_i\left(\mathbb{k}\right)$ is small
• IF $e_i\left(\mathbb{k}\right)$ is large, and ${\dot{q}}_i\left(\mathbb{k}\right)$ is large THEN ${\lambda }_i\left(\mathbb{k}\right)$ is large
• IF $e_i\left(\mathbb{k}\right)$ is large, and ${\dot{q}}_i\left(\mathbb{k}\right)$ is small THEN ${\lambda }_i\left(\mathbb{k}\right)$ is large
• IF $e_i\left(\mathbb{k}\right)$ is small, and ${\dot{q}}_i\left(\mathbb{k}\right)$ is large THEN ${\lambda }_i\left(\mathbb{k}\right)$ is small

Remark 8. 

To carry out the linguistic inference shown by (32) in a numerical form, fuzzy logic inference is required. In detail, initially, numerical variables $e_i\left(\mathbb{k}\right)$ and ${\dot{q}}_i\left(\mathbb{k}\right)$ at the moment $\mathbb{k}$ are fuzzified to the firing rates of the linguistic descriptions by the triangular membership function shown in Figure 2. After that, a group of firing rates of fuzzy rules are obtained by fuzzy reasoning. Then, the numerical values of ${\lambda }_i\left(\mathbb{k}\right)$ are calculated by the defuzzification according to the firing rates of all fuzzy rules and the numerical value of the action corresponding to each fuzzy rule.

The parameters of membership function shown in Figure 2 are defined as follows:

$$Lin\left(e_i\right)=\left\{{\zeta }_{1,1}^{\left(i\right)},\dots , {\zeta }_{1,𝒶}^{\left(i\right)},\dots ,{\zeta }_{1,\mathcal{A}}^{\left(i\right)}\right\}, 𝒶=1,2,\dots ,\mathcal{A}$$

(33)

$$Lin\left({\dot{q}}_i\right)=\left\{{\zeta }_{2,1}^{\left(i\right)},\dots , {\zeta }_{2,𝒷}^{\left(i\right)},\dots ,{\zeta }_{2,ℬ}^{\left(i\right)}\right\}, 𝒷=1,2,\dots ,ℬ$$

(34)

where $\mathcal{A}$ is the number of fuzzy sets (${\zeta }_{1,𝒶}$) for the fuzzy input $e_i$, and $ℬ$ is the number of fuzzy sets (${\zeta }_{1,𝒷}$) for the fuzzy input ${\dot{q}}_i$.

The $𝓃$-th fuzzy rule in fuzzy Q learning is defined as follows:

$$\begin{gathered} R_{𝓃,i}: \mathrm{IF} {𝓈}_{1,i}^{\left(\mathbb{k}\right)} \mathrm{is} L_{1,i}^{\left(\mathbb{k}\right)} \mathrm{and} {𝓈}_{2,i}^{\left(\mathbb{k}\right)} \mathrm{is} L_{2,i}^{(𝓃)} \mathrm{and} \dots \dots \mathrm{and} {𝓈}_{𝓂,i}^{\left(\mathbb{k}\right)} \mathrm{is} L_{𝓂,i}^{(𝓃)}, \mathrm{THEN} {\mathfrak{u}}_i^{(𝓃)}\in {\mathfrak{U}}_{i,𝓃} \mathrm{that} {\mathfrak{u}}_i^{(𝓃)}={\mathfrak{u}}_{i,1}^{(𝓃)} \mathrm{with} \\ {𝓆}_i^{\left(k\right)}\left(𝓃,1\right) \mathrm{or} {\mathfrak{u}}_i^{(𝓃)}={\mathfrak{u}}_{i,2}^{(𝓃)} \mathrm{with} {𝓆}_i^{\left(k\right)}\left(𝓃,2\right) \mathrm{or} {\mathfrak{u}}_i^{(𝓃)}={\mathfrak{u}}_{i,3}^{(𝓃)} \mathrm{with} {𝓆}_i^{\left(k\right)}\left(𝓃,3\right),\dots ., {\mathfrak{u}}_i^{(𝓃)}={\mathfrak{u}}_{i,p}^{(𝓃)} \mathrm{with} \\ {𝓆}_i^{\left(k\right)}\left(𝓃,p\right),\dots .,{\mathfrak{u}}_i^{(𝓃)}={\mathfrak{u}}_{i,P}^{(𝓃)}. \mathrm{with} {𝓆}_i^{\left(k\right)}\left(𝓃,P\right) \end{gathered}$$

(35)

where ${\mathfrak{U}}_{i,𝓃}=\left\{{\mathfrak{u}}_{i,1}^{(𝓃)},..{\mathfrak{u}}_{i,p}^{(𝓃)},..,{\mathfrak{u}}_{i,P}^{(𝓃)}\right\}$ is the set of action candidates in the rule $R_{𝓃,i}$. $L_i^{(𝓃)}=${$L_{1,i}^{(𝓃)},\dots ,L_{𝓂,i}^{(𝓃)}$} is the set of linguistic variables of fuzzy inputs. ${𝓈}_i^{\left(\mathbb{k}\right)}=\left\{{𝓈}_{1,i}^{\left(\mathbb{k}\right)},\dots , {𝓈}_{𝓂,i}^{\left(\mathbb{k}\right)}\right\}$ is the set of fuzzy inputs at the $\mathbb{k}$ moment. In this paper, the fuzzy inputs are $e_i$ and ${\dot{q}}_i$ such that ${𝓈}_i^{\left(\mathbb{k}\right)}=\left\{e_i\left(\mathbb{k}\right), {\dot{q}}_i\left(\mathbb{k}\right)\right\}$.

The set of fuzzy inputs ${𝓈}_i^{\left(\mathbb{k}\right)}=\left\{{𝓈}_{1,i}^{\left(\mathbb{k}\right)},\dots , {𝓈}_{𝓂,i}^{\left(\mathbb{k}\right)}\right\}$ is fuzzified by the membership function shown in Figure 2 and then matched with the rule antecedents (19), providing the firing rate vector $\phi \left({𝓈}_i^{\left(\mathbb{k}\right)}\right)=\left[{\phi }_1\left({𝓈}_i^{\left(\mathbb{k}\right)}\right),{\phi }_2\left({𝓈}_i^{\left(\mathbb{k}\right)}\right),..,{\phi }_{\mathcal{N}}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)\right]$. $\mathcal{N}$ is the amount of fuzzy rules (There are $\mathcal{N}$ fuzzy rules).

For the ${𝓃}^{th}$ rule in the $i^{th}$ subsystem ($R_{𝓃,i}$), the optimal action at the $\mathbb{k}$ moment is defined as the action with the maximum ${𝓆}_i^{\left(\mathbb{k}\right)}\left(𝓃,p\right)$, $p\in \left\{1,2,..,P\right\}$ among $P$ action candidates.

$${\mathfrak{u}}^{*}{}_i^{(𝓃)}=arg\underset{{\mathfrak{u}}_i^{(𝓃)}\in {\mathfrak{U}}_{i,𝓃}}{\max }{𝓆}_i^{\left(\mathbb{k}\right)}\left(𝓃,{\mathfrak{u}}_{i,p}^{(𝓃)}\right)$$

(36)

To prevent the selection of $\mathfrak{u}$ from the local optimum in the learning process, we introduce a greed mechanism:

$${\widehat{\mathfrak{u}}}_i^{(𝓃)}=\{\begin{array}{c}{\mathfrak{u}}^{+}{}_i^{(𝓃)}, with probablity \rho \\ {\mathfrak{u}}^{*}{}_i^{(𝓃)} with probablity 1-\rho \end{array}$$

(37)

where ${\mathfrak{u}}^{+}{}_i^{(𝓃)}$ is an action randomly selected from ${\mathfrak{U}}_{i,𝓃}$. $0<\mathsf{\rho }<1$ is the probability to explore random actions.

The numerical value of ${\lambda }_i$ is calculated by firing rates and the selected actions:

$${\lambda }_i\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)=\frac{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right){\widehat{\mathfrak{u}}}_i^{(𝓃)}}{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)}$$

(38)

The update principle of $𝓆$ is the most important part in the whole learning process. The $𝓆$-values are updated according to the rewards of the selected actions; the optimal action can achieve higher rewards, therefore, we can finally learn the optimal action.

The Q value at the $\mathbb{k}$ moment can be designed as follows:

$$Q\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)=\frac{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right){𝓆}_i^{\left(\mathbb{k}\right)}\left(𝓃, {\widehat{\mathfrak{u}}}_i^{(𝓃)}\right)}{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)}$$

(39)

The target value at the state ${𝓈}_i^{\left(k\right)}$ is calculated as:

$$\mathcal{V}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)=\frac{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right){𝓆}_i^{\left(\mathbb{k}\right)}\left(𝓃, {\mathfrak{u}}^{*}{}_i^{(𝓃)}\right)}{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)}$$

(40)

When the system state ${𝓈}_i^{\left(\mathbb{k}\right)}$ is driven to the next state ${𝓈}_i^{\left(\mathbb{k}+1\right)}$, temporal difference (TD) is calculated according to the reward obtained at the $k$ moment ${𝓇}_i^{\left(\mathbb{k}\right)}$:

$$∆Q_i^{\left(\mathbb{k}\right)}={𝓇}_i^{\left(\mathbb{k}\right)}+{\alpha }_i\mathcal{V}\left({𝓈}_i^{\left(\mathbb{k}+1\right)}\right)-Q\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)$$

(41)

where ${\alpha }_i\in \left[0,1\right]$ is the discount factor reflecting the contribution of the future reward. The reward ${𝓇}_i^{\mathbb{k}}$ at the $\mathbb{k}$ moment of the $i^{th}$ subsystem is designed in (42) and the meaning of (42) is explained in Remark 9.

$${𝓇}_i^{\left(\mathbb{k}\right)}=\{\begin{array}{c}0, if \left|{𝓍}_i\left(\mathbb{k}\right)\right|>{\sigma }_i\\ \cos \left(\frac{\pi }{2}\frac{\left|{𝓍}_i\left(\mathbb{k}\right)\right|}{{\sigma }_i}\right), if \left|{𝓍}_i\left(\mathbb{k}\right)\right|\le {\sigma }_i \end{array}$$

(42)

where ${𝓍}_i\left(\mathbb{k}\right)={\dot{q}}_i\left(\mathbb{k}\right)+{\Lambda }_{i}tanh\left(\frac{e_i\left(\mathbb{k}\right)}{{₵}_i}\right)$ and ${\sigma }_i>0$ is a positive constant.

Finally, the adaptive law of q-values is:

$${𝓆}_i^{\left(\mathbb{k}+1\right)}\left(𝓃,{\widehat{\mathfrak{u}}}_i^{(𝓃)}\right)={𝓆}_i^{\left(\mathbb{k}\right)}\left(𝓃,{\widehat{\mathfrak{u}}}_i^{(𝓃)}\right)+{\gamma }_i·∆Q_i^{\left(\mathbb{k}\right)}·\frac{{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)}{{\sum }_{𝓃=1}^{\mathcal{N}}{\phi }_{𝓃}\left({𝓈}_i^{\left(\mathbb{k}\right)}\right)}$$

(43)

where ${\gamma }_i\in \left[0,1\right]$ is the learning rate.

Remark 9. 

The reward function (42) indicates that higher values of reward ${𝓇}_i^{\left(\mathbb{k}\right)}$ can be obtained by the smaller ${𝓍}_i\left(\mathbb{k}\right)$, which means the the action that satisfies (24) will be given with the highest reward (${𝓇}_i^{\left(\mathbb{k}\right)}=1$).

The aforementioned process of fuzzy Q learning to tune ${\lambda }_i$ can be also concluded in Figure 3. The proposed control scheme can be concluded in Figure 4.

Remark 10. 

All the system states including the tracking errors are proven to be bounded with ${\lambda }_i>0$. Therefore, the instability will not occur even if an inappropriate positive value of ${\lambda }_i$ is calculated by the fuzzy Q learning mechanism trying some bad action candidates. Hence, the designed control law (7), (9)–(14) offers the fuzzy Q learning mechanism (32)–(43) a safe environment to learn a optimal ${\lambda }_i$. Notably, in order to make sure ${\lambda }_i>0$ always holds, all the action candidates ${\mathfrak{U}}_{i,𝓃}=\left\{{\mathfrak{u}}_{i,1}^{(𝓃)},..{\mathfrak{u}}_{i,p}^{(𝓃)},..,{\mathfrak{u}}_{i,P}^{(𝓃)}\right\}$ should be positive.

Remark 11. 

The proof of Theorem 1 requires the parameter ${\lambda }_i>0$ to be a constant. However, the parameter ${\lambda }_i$ is oneline tuned by a fuzzy Q learning mechanism, and therefore ${\lambda }_i$ is varying in the implementation of the proposed controller. To handle this issue, we design that the time interval between two consecutive fuzzy inferences of ${\lambda }_i$ in fuzzy Q learning is 20-times greater than the time interval between two consecutive control actions $\tau$ (eg. 0.1s between two consecutive ${\widehat{\mathfrak{u}}}_i^{(𝓃)}$ (equally, two consecutive ${\lambda }_i$) while 0.001s between two consecutive $\tau$). Namely, ${\lambda }_i\left(t\right)={\lambda }_i\left(t^{*}\right), \forall t\in \left[t^{*},t^{*}+20L\right]$; $\tau \left(t\right)=$ $\tau \left(t^{*}\right)$, $\forall t\in \left[t^{*},t^{*}+L\right]$. $t^{*}$ is any time moment when the algorithm works. Thereby, not only can the time derivative of ${\lambda }_i$ be negligible in the proof of Theorem 1, but the computational load is also decreased.

4. Simulation Results and Analysis

In this section, similar to works [38,39,40,41,42], we use the simulation to verify the effectiveness of the proposed controller. A 2-rigid-link robotic manipulator shown in Figure 5 is carried in Matlab 2018a. The sampling time of simulating the real dynamics of robotic manipulators is set as $1\times {10}^{-4}$ s. The sampling time of TDE and controller is set as $1\times {10}^{-3}$ s (10 times to $1\times {10}^{-4}$) to show the discrete nature of using controllers in practice. The sampling time of fuzzy Q learning is set as 0.01 s according to Remark 10. The dynamic model of a 2-rigid-link robotic manipulator is given as follows, which can be also found in [26].

$$\mathit{M}\left(\mathit{q}\right)=\left[\begin{array}{cc}{m}_2{l}_2^2+2l_1{l}_2{m}_2\cos \left(q_2\right)+\left(m_1+m_2\right)l_1^2& m_2{l}_2^2+l_1{l}_2{m}_2\cos \left(q_2\right)\\ m_2{l}_2^2+l_1{l}_2{m}_2\cos \left(q_2\right)& m_2{l}_2^2\end{array}\right]$$

$$\mathit{C}\left(\mathit{q},\dot{\mathit{q}}\right)\dot{\mathit{q}}=\left[\begin{array}{c}-m_2{l}_1{l}_2\sin \left(q_2\right){\dot{q}}_2^2-2m_2{l}_1{l}_2\sin \left(q_2\right){\dot{q}}_1{\dot{q}}_2\\ m_2{l}_1{l}_2\sin \left(q_2\right){\dot{q}}_1^2\end{array}\right]$$

$$\mathit{G}\left(\mathit{q}\right)=\left[\begin{array}{c}\left(m_1+m_2\right)l_{1}cos\left(q_2\right)g+m_2{l}_{2}cos\left(q_1+q_2\right)g\\ m_2{l}_2\cos \left(q_1+q_2\right)g\end{array}\right]$$

$$\mathit{F}\left(\dot{\mathit{q}}\right)=\left[\begin{array}{c}{F}_1\\ F_2\end{array}\right]$$

$${\mathit{\tau }}_{\mathit{d}}=\left[\begin{array}{c}{\tau }_{d1}\\ {\tau }_{d2}\end{array}\right]$$

The system parameters are given in

Table 1. Parameters of robotic manipulator.

Robotic manipulator	Parameter	Value
	$l_1$ ($m$)	0.5
	$l_2$ ($m)$	0.5
	$m_1$ ($kg)$	5
	$m_2$ ($kg)$	2
	$g$ ($m/s^2$)	9.8

, which are the same as [26].

The initial angular positions and velocities of two joints are set as: $q_1=0^{\mathrm{o}}$, ${\mathrm{q}}_2=0^{\mathrm{o}}$, ${\dot{q}}_1=0^{\mathrm{o}}/s$ and ${\dot{q}}_2=0^{\mathrm{o}}/\mathrm{s}$. The desired angular positions of two joints are set as: $q_{r1}={30}^{\mathrm{o}}$ and $q_{r2}={45}^{\mathrm{o}}$.

The upper bounds of the angular position and angular velocity for the $i^{th}$ joint of manipulator (${\epsilon }_i$ and ${\Lambda }_i$) are selected by the user dependent on the specific mission requirement. To successfully implement the proposed controller, users can select any values satisfying ${\epsilon }_i>\left|q_{ri}\right|$ and ${\Lambda }_i>0$. Therefore, in the simulation of this paper, we select ${\epsilon }_1={50}^{\mathrm{o}}$, ${\epsilon }_2={60}^{\mathrm{o}}, {\Lambda }_1={10}^{\mathrm{o}}/\mathrm{s}$ and ${\Lambda }_2={12}^{\mathrm{o}}/s$.

Remark 12. 

$k_{pi}$ and $k_{di}$ are the proportional coefficient and differential coefficient, respectively. The great $k_{pi}$ could decrease steady state error, but an excessively great could result in a significant overshoot. While the great $k_{di}$ could improve the robustness to disturbance/uncertainty, but an over-great $k_{di}$ could compromise the tracking accuracy. The selection of $k_{pi}$ and $k_{di}$ can be based on the tuning rules of PID controllers.

Remark 13. 

Great $k_{si}$ can amplify the effect of the terms $A_i$ and $B_i$ in (10) to keep the angular positions from hitting their constraints. An inappropriately small $k_{si}$ could lead to a $\left|q_i\right|$ too close to ${\epsilon }_i$ and then result in an over-great $u_i$ in (10). Therefore, the selection of $k_{si}$ could be started at a small value, and then users could gradually increase $k_{si}$ until the magnitude of $u_i$ is acceptable. Great $k_{yi}$ can amplify the effect of the term $C_i$ in (10) to keep the angular velocities from hitting their constraints. An excessively great $k_{yi}$ could lead to an over great ($1+k_{yi}{C}_i)$ in (10) to decline the converging rate of the tracking error even if the angular velocity is not close to its constraint, which could increase the settling time. Therefore, the trial of selecting $k_{yi}$ could start at a small value, and then users could decrease it until the converging rate of the tracking error is acceptable. ${\beta }_i$ determines the converging rate of the auxiliary variable $z_i$ that has a significant effect on steady state error in the controller [33]. However, in the proposed controller, the RL-based term is introduced to improve the tracking performance and thereby the effect of ${\beta }_i$ is decreased. We suggest to offer ${\beta }_i$ with a medium value ranging between 0.1–1.

Remark 14. 

large values of ${\psi }_i$ and ${\delta }_i$ can lead to a fast adaption of ${\widehat{d}}_i$ to handle system uncertainty and external disturbance. However, the inappropriately large ${\psi }_i$ and ${\delta }_i$ could result in a sharp variation of ${\widehat{d}}_i$ and then a chattering effect on control torques. Therefore, the selection of ${\psi }_i$ and ${\delta }_i$ could start at a big value, and then users could decrease them until no chattering effect occurs on the control torques. Over-small values of ${\overline{\Omega }}_i$ could lead to an insufficient decrease of ${\widehat{d}}_i$ that still brings up a chattering effect on control torques. Meanwhile, over-great values of ${\overline{\Omega }}_i$ could negatively influence the robustness to uncertainty and disturbance. Hence, the selection of ${\overline{\Omega }}_i$ could start at some small values, and then users could increase ${\overline{\Omega }}_i$ until ${\widehat{d}}_i$ is significantly decreased at the final stage of the control to have a satisfactory chattering attenuation.

Remark 15. 

Small/great values of ${₵}_i$ can amplify/reduce the effect of the RL term ${\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)$ in (10). The over-small values of ${₵}_i$ could result in the over-great magnitude of $u_i$, while the over-great values of ${₵}_i$ could bring up an insufficient improvement on tracking performance. Therefore, the trial of selecting ${₵}_i$ is suggested to start at a great value, and then users could decrease ${₵}_i$ until a satisfying improvement on the tracking performance.

According to Remark 11~Remark 15, the parameters of the proposed controller are selected in

Table 2. Parameters of proposed controller.

Parameter	Value $(\mathit{i}=1)$	Value $(\mathit{i}=2)$
$k_{pi}$	10	10
$k_{di}$	1	1
$k_{si}$	0.1	0.1
${₵}_i$	0.1	0.1
$k_{yi}$	0.1	0.1
${\beta }_i$	0.1	0.1
${\psi }_i$	0.01	0.01
${\overline{\Omega }}_i$	0.05	0.05
${\delta }_i$	0.1	0.1

.

Notably, it does not require an extensive trial to select parameters in Table 2 as the satisfying tracking performance is mainly obtained by the optimal ${\lambda }_i$ in the RL term ${\lambda }_i{\Lambda }_{i}tanh\left(\frac{e_i }{{₵}_i}\right)$. In other words, users can select some medium values for parameters (not optimal) in Table 2 by an acceptable amount of parameters-selection trails. Then, the fuzzy Q learning can automatically explore the ${\lambda }_i$ matching the selected parameters to have a satisfying tracking performance.

Remark 16. 

Similar to [36], the parameters of membership function in (33) and (34), $Lin\left(e_i\right)=\left\{{\zeta }_{1,1}^{\left(i\right)}, {\zeta }_{1,2}^{\left(i\right)},\dots ,{\zeta }_{1,\mathcal{A}}^{\left(i\right)}\right\}$ and $Lin\left({\dot{q}}_i\right)=\left\{{\zeta }_{2,1}^{\left(i\right)}, {\zeta }_{2,2}^{\left(i\right)},\dots ,{\zeta }_{2,ℬ}^{\left(i\right)}\right\}$, are used to fuzzify $e_i$ and ${\dot{q}}_i$. To well present the $e_i$ and ${\dot{q}}_i$ in the form of firing rate, we suggest to offer ${\zeta }_{1,1}^{\left(i\right)}$ and ${\zeta }_{2,1}^{\left(i\right)}$ with some small values and provide some great values to ${\zeta }_{1,\mathcal{A}}^{\left(i\right)}$ and ${\zeta }_{2,ℬ}^{\left(i\right)}$. The ${\zeta }_{1,𝒶}^{\left(i\right)}$ and ${\zeta }_{2,𝒷}^{\left(i\right)}$ are suggested to be evenly distributed among (${\zeta }_{1,1}^{\left(i\right)}$,${\zeta }_{1,\mathcal{A}}^{\left(i\right)}$) and (${\zeta }_{2,1}^{\left(i\right)}$,${\zeta }_{2,ℬ}^{\left(i\right)}$), respectively. The great $\mathcal{A}$ and $ℬ$ increase the potential to well present $e_i$ and ${\dot{q}}_i$ at the cost of increasing the computational load. Therefore, the trial of selecting $\mathcal{A}$ and $ℬ$ could start at some great values (e.g., 20), and then users could decrease them until a reasonable computational load.

Remark 17. 

${\sigma }_i$ is the threshold of obtaining rewards. The inappropriately small values of ${\sigma }_i$ could result in a difficulty in obtaining high values of reward, while the over-great values of ${\sigma }_i$ could result in the different action candidates to be offered with the similarly high rewards even if they lead to the different control performances. Therefore, both of the over-great and over-small values of ${\sigma }_i$ will negatively influence the convergence of q-values in (43) and thereby compromise the performance of reinforcement learning. The selection of ${\sigma }_i$ could start at a small value, and the users could increase ${\sigma }_i$ until a satisfying convergence of q values (the convergence of q-values can be also reflected by the convergence of obtained rewards). The selection of mutation probability $\rho$, learning rate ${\gamma }_i$ and discount factor ${\alpha }_2$ can be based on the strategy mentioned in [36].

Remark 18 

Action candidates ${\mathfrak{U}}_{i,𝓃}=\left\{{\mathfrak{u}}_{i,1}^{(𝓃)},..{\mathfrak{u}}_{i,p}^{(𝓃)},..,{\mathfrak{u}}_{i,P}^{(𝓃)}\right\}$ are the most important parameters in the proposed control scheme because the optimal ${\lambda }_i$ that brings up a satisfying control performance is calculated based on them. To make the optimal action included in the group of action candidates, ${\mathfrak{u}}_{i,1}^{(𝓃)}$ should be given a small value (e.g., 0) while ${\mathfrak{u}}_{i,P}^{(𝓃)}$ should be given a great value. The rest candidates ${\mathfrak{u}}_{i,p}^{(𝓃)}$ ($p=1,2,..,p$) are suggested to be evenly distributed between ${\mathfrak{u}}_{i,1}^{(𝓃)}$ and ${\mathfrak{u}}_{i,P}^{(𝓃)}$. Users could initially give ${\mathfrak{u}}_{i,P}^{(𝓃)}$ with a small value (but greater than ${\mathfrak{u}}_{i,1}^{(𝓃)}$), and an increase of ${\mathfrak{u}}_{i,P}^{(𝓃)}$ until a sufficient improvement on the tracking performance is achieved. $p$ is the amount of action candidates of each fuzzy rule for each subsystem. The selection of $p$ could start at a great number (e.g., 20), and then users could decrease $p$ until the computational load is acceptable.

According to Remark 16~Remark 18, the parameters of the fuzzy Q learning mechanism are given as follows. The amount of fuzzy sets in (17) are $\mathcal{A}=ℬ=8$. Therefore, the amount of fuzzy rules are $\mathcal{N}=64$. The parameters of membership function to do the fuzzification are: $Lin\left(e_1\right)=\left\{{\zeta }_{1,1}^{\left(1\right)}, {\zeta }_{1,2}^{\left(1\right)},\dots ,{\zeta }_{1,8}^{\left(1\right)}\right\}=${−0.0017, −0.0014, −0.001, −0.0006, −0.0002, 0.0002, 0.0006, 0.001, 0.0014, 0.0017}. $Lin\left({\dot{q}}_1\right)=\left\{{\zeta }_{2,1}^{\left(1\right)}, {\zeta }_{2,2}^{\left(1\right)},\dots ,{\zeta }_{2,8}^{\left(1\right)}\right\}=${−0.175, −0.138, −0.097, −0.058, −0.019, 0.019, 0.058, 0.097, 0.138, 0.175}$\times {10}^{-3}$. $Lin\left(e_2\right)=\left\{{\zeta }_{1,1}^{\left(2\right)}, {\zeta }_{1,2}^{\left(2\right)},\dots ,{\zeta }_{1,8}^{\left(2\right)}\right\}=${−0.0017, −0.0014, −0.001, −0.0006, −0.0002, 0.0002, 0.0006, 0.001, 0.0014, 0.0017}. $Lin\left({\dot{q}}_2\right)=\left\{{\zeta }_{2,1}^{\left(2\right)}, {\zeta }_{2,2}^{\left(2\right)},\dots ,{\zeta }_{2,8}^{\left(2\right)}\right\}=${−0.175, −0.138, −0.097, −0.058, −0.019, 0.019, 0.058, 0.097, 0.138, 0.175}$\times {10}^{-3}$. The action candidates for each fuzzy rule are: ${\mathfrak{U}}_{i,𝓃}=\left\{0,22,44,66,88,111,133,155,177,200\right\}$ for all $i=1,2$ and $𝓃=1,2,\dots ., 64$. Therefore, the amount of action candidates for each fuzzy rule is $P=10$. The discount factor is ${\alpha }_1={\alpha }_2=0.01$ and the learning rate is ${\gamma }_1={\gamma }_2=0.2$. The threshold of obtaining rewards is ${\sigma }_1={\sigma }_2=0.01$.

To show the superiority, four existing controllers, Refs. [26,33,43,44] are compared with the proposed controller.

The controller from [26] is given as (44)–(52)

$$\mathit{\tau }={\mathit{\tau }}_{\mathbf{1}}+{\mathit{\tau }}_{\mathbf{2}}$$

(44)

$${\mathit{\tau }}_{\mathbf{1}}=\widehat{\mathit{M}}\left(\mathit{q}\right)\left({\ddot{\mathit{q}}}_{\mathit{r}}-{\mathit{K}}_{\mathit{V}}\dot{\mathit{e}}-{\mathit{K}}_{\mathit{P}}\mathit{e}\right)+\widehat{\mathit{C}}\left(\mathit{q},\dot{\mathit{q}}\right)\dot{\mathit{q}}+\widehat{\mathit{G}}\left(\mathit{q}\right)$$

(45)

$${\mathit{\tau }}_{\mathbf{2}}=-\widehat{\mathit{M}}\left(\mathit{q}\right)\widehat{\mathit{f}}+\widehat{\mathit{M}}\left(\mathit{q}\right){\mathit{u}}_{\mathit{r}}$$

(46)

$${\mathit{u}}_{\mathit{r}}=\left[\begin{array}{c}-\widehat{\xi }\tan \mathrm{h}\left(a_{1}p/{\rho }_1\right)\\ -\widehat{\xi }\tan \mathrm{h}\left(a_{2}p/{\rho }_1\right)\end{array}\right]$$

(47)

$$\widehat{\mathit{f}}={\widehat{\mathit{W}}}^{\mathit{T}}\mathit{\sigma }\left({\widehat{\mathit{V}}}^{\mathit{T}}\mathit{X}\right)$$

(48)

$$\dot{\widehat{\xi }}=a_{1}ptanh\left(\frac{a_{1}p}{{\rho }_1}\right)+a_{1}ptanh\left(\frac{a_{2}p}{{\rho }_1}\right)-K\widehat{\xi }$$

(49)

$$\dot{\widehat{W}}=\left(\mathit{\sigma }-{\mathit{\sigma }}^{\prime }{\widehat{V}}^{T}X\right)x^{T}PB-{\mathsf{{\rm Y}}}_W\widehat{W}$$

(50)

$$\dot{\widehat{V}}=X{x}^{T}PB{\widehat{W}}^T{\mathit{\sigma }}^{\prime }-{\mathsf{{\rm Y}}}_V\widehat{V}$$

(51)

$$p=1+\left|\left|X\right|\right|+\left|\left|\widehat{V}\right|\right|·\left|\left|X\right|\right|+\left|\left|\widehat{W}\right|\right|·\left|\left|X\right|\right|$$

(52)

where $\mathit{\sigma }={[{\sigma }_1,{\sigma }_2,\dots ,{\sigma }_m]}^T$ is the vector of hidden neurons with the activation function ${\sigma }_i\left(s\right)=1/\left(1+e^{-s}\right)$. ${\mathit{\sigma }}^{\prime }$ is the vector of partial derivative of $\sigma$ such that ${\mathit{\sigma }}^{\prime }=\partial \mathit{\sigma }\left(s\right)/\partial s$. $x={[e_1,e_2,{\dot{q}}_1,{\dot{q}}_2]}^T$ and $X={[q_{r1},q_{r2},q_1,q_2,{\dot{q}}_1,{\dot{q}}_2,{\ddot{q}}_1,{\ddot{q}}_2]}^T$. $P$ satisfies $PA+A^{T}P=-Q$ with $A=\left[\begin{array}{cc}0& I\\ -K_P& -K_V\end{array}\right]$. $\left[a_1,a_2\right]=x^{T}PB$.

In this simulation, we let the controller from [26] to fully know the system parameters such that $\widehat{\mathit{M}}\left(\mathit{q}\right)=\mathit{M}\left(\mathit{q}\right)$, $\widehat{\mathit{C}}\left(\mathit{q},\dot{\mathit{q}}\right)=\mathit{C}\left(\mathit{q},\dot{\mathit{q}}\right)$ and $\widehat{\mathit{G}}\left(\mathit{q}\right)=\mathit{G}\left(\mathit{q}\right)$.

The parameters in (44)–(52) are given as:

$$K_V=\left[\begin{array}{cc}300& 0\\ 0& 300\end{array}\right], K_P=\left[\begin{array}{cc}200& 0\\ 0& 200\end{array}\right]$$

$$B=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 1& 0\\ 0& 1\end{array}\right], Q=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

${\rho }_1=0.01$, $K=0.005$, ${\mathsf{{\rm Y}}}_W=0.15$, ${\mathsf{{\rm Y}}}_V=0.15$, $\widehat{\xi }\left(0\right)=0.01$

$$\widehat{W}\left(0\right)=\left[\begin{array}{cc}-0.1& -0.1\\ -0.1& -0.1\\ -0.1& -0.1\\ -0.1& -0.1\\ -0.1& -0.1\end{array}\right]$$

$$\widehat{V}\left(0\right)=\left[\begin{array}{ccccc}0.1& 0.1& 0.1& 0.1& 0.1\\ 0.1& 0.1& 0.1& 0.1& 0.1\\ 0.1& 0.1& 0.1& 0.1& 0.1\\ 0.1& 0.1& 0.1& 0.1& 0.1\\ 0.1& 0.1& 0.1& 0.1& 0.1\\ 0.1& 0.1& 0.1& 0.1& 0.1\end{array}\right]$$

The controller from [33] is given as (53)–(58).

$$\tau =\mathcal{G}\left(q\right)\widehat{p}-K_d\dot{q}-K_{p}e-K_{s}diag\left\{{\mathcal{X}}_1,{\mathcal{X}}_2\right\}z$$

(53)

$${\mathcal{X}}_i=\frac{{\zeta }_i^2}{{\zeta }_i^2-q_i^2}+\frac{{\zeta }_i^2{q}_i{z}_i}{{({\zeta }_i^2-q_i^2)}^2}, i=1,2$$

(54)

$$z_i=e_i+{\int }_0^t{\eta }_i(\mathsf{\Theta })d\mathsf{\Theta }, i=1,2$$

(55)

$${\eta }_i=-{\beta }_i{z}_i, i=1,2$$

(56)

$$\dot{\widehat{p}}=-\mathsf{\Psi }{\mathcal{G}}^T\left(q\right)\dot{q}$$

(57)

$$\mathcal{G}\left(q\right)=\left[\begin{array}{cc}\cos \left(q_1+q_2\right)& \cos \left(q_2\right)\\ \cos \left(q_1+q_2\right)& 0\end{array}\right]$$

(58)

where $\widehat{p}={[{\widehat{p}}_1,{\widehat{p}}_2]}^T$, $z_1\left(0\right)=e_1\left(0\right)$ and $z_2\left(0\right)=e_2\left(0\right)$. ${\zeta }_1$ and ${\zeta }_2$ are the specific constraint (designed upper bound) of the 1st and 2nd joint respectively, therefore, ${\zeta }_1={\epsilon }_1={50}^{\mathrm{o}}$ and ${\zeta }_2={\epsilon }_2={60}^{\mathrm{o}}$.

The parameters in (53)–(58) are given as follows:

$$K_d=\left[\begin{array}{cc}60& 0\\ 0& 20\end{array}\right], K_p=\left[\begin{array}{cc}24& 0\\ 0& 24\end{array}\right], K_s=\left[\begin{array}{cc}20& 0\\ 0& 20\end{array}\right]$$

$${\beta }_1=30, {\beta }_2=30, \mathsf{\Psi }=\left[\begin{array}{cc}0.1& 0\\ 0& 0.1\end{array}\right]$$

The controller from [43] is given as (59)–(61).

$$\tau =K_D^{-1}\widehat{M }\left[K_P\dot{e}-K_{I}e+K_D{\ddot{q}}_r+\beta \dot{s}-K_D\widehat{\Gamma }-K_D\lambda s-K_D{k}_{s}sgn\left(\dot{s}\right)\right]$$

(59)

$$\dot{s}+\beta s=K_{P}e+K_I{\int }_0^{t}e+K_D\dot{e}$$

(60)

$${\dot{k}}_s=\gamma \left|\left|\dot{s}\right|\right|$$

(61)

where $\dot{s}\left(0\right)={[0,0]}^T$, $k_s\left(0\right)=0$, $\gamma =0.1$, $\widehat{\Gamma }$ is the known part of lumped uncertainty $\Gamma$. In this simulation, we let $\widehat{\Gamma }=0.7\Gamma$. The parameters $K_P$, $K_D$, $K_I$ and $\beta$ are given as follows.

$$K_D=\left[\begin{array}{cc}20& 0\\ 0& 20\end{array}\right], K_P=\left[\begin{array}{cc}10& 0\\ 0& 10\end{array}\right], K_I=\left[\begin{array}{cc}0.01& 0\\ 0& 0.01\end{array}\right], \beta =\left[\begin{array}{cc}0.1& 0\\ 0& 0.1\end{array}\right]$$

The controller from [44] is given as (62)–(65).

$$\{\begin{array}{c}\widehat{D}=z+Y\dot{q}\\ \dot{z}=-Y{\widehat{M}}^{-1}z+Y{\widehat{M}}^{-1} \left(-\tau -Y\dot{q}-\overline{D}\right)\end{array}$$

(62)

$$\tau =\widehat{M}\left({\ddot{q}}_r-\mathsf{\Lambda }\dot{q}+\mathsf{\Lambda }{\dot{q}}_r\right)-\overline{D}-K_{D}sat\left(s\right)-\widehat{D}$$

(63)

$$sat\left(s_i\right)=\{\begin{array}{c}sgn\left(s_i\right), \left|s_i\right|\ge \sigma \\ s_i/\sigma ,\left|s_i\right|<\sigma \end{array}$$

(64)

where $\sigma =0.1$, $sat\left(s\right)={[s_1,s_2]}^T$. $z\left(0\right)={[0,0]}^T$ and $s=\dot{e}+\mathsf{\Lambda }e$. $\overline{D}$ is known part of ${\widehat{M}}^{-1}\mathsf{\Gamma }$, $\overline{D}=0.7{\widehat{M}}^{-1}\mathsf{\Gamma }$ in the simulation. The parameters $Y$, $\mathsf{\Lambda }$ and $K_D$ are given as follows.

$$K_D=\left[\begin{array}{cc}40& 0\\ 0& 16\end{array}\right], Y=\left[\begin{array}{cc}\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$0.06$}\right.& 0\\ 0& \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$0.06$}\right.\end{array}\right], \mathsf{\Lambda }=\left[\begin{array}{cc}40& 0\\ 0& 20\end{array}\right]$$

Furthermore, two cases are considered to run the simulation. In the 1st case, the parameters of the dynamic model are fully known, and there is no external disturbance and no friction. In the 2nd case, the parametric uncertainty is considered and the unknown external disturbance and frictions are applied on the dynamics model.

Remark 19. 

The selections of parameters for the proposed controller, the controller from [26] and the controller from [33] are all carried out in case 1. In other words, all the parameters for the three different controllers are fine-tuned to have a good performance in case 1. In case 2, all the selected parameters of the three controllers remain unchanged to test the robustness to the lumped uncertainty.

Case 1. 

In the absence of system uncertainty, external disturbance and friction.

In this case, the system parameters used in the three controllers (proposed controller, controller from [26] and controller from [33]) are the same as the parameters in the dynamics model of manipulator, which means no parametric uncertainty. Meanwhile, the friction and external disturbance are null, shown as follows:

$$\mathit{F}\left(\dot{\mathit{q}}\right)=\left[\begin{array}{c}{F}_1\\ F_2\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right], {\mathit{\tau }}_{\mathit{d}}=\left[\begin{array}{c}{\tau }_{d1}\\ {\tau }_{d2}\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

The comparisons of angular position tracking are shown in Figure 6 and Figure 7, while the comparisons of angular velocity are shown in Figure 8 and Figure 9. Cleanly, in the absence of unknown disturbance and system parametric uncertainty, the proposed controller shows an inferior performance in terms of a greater steady state error and a slower error converging rate, compared to the controllers from [26,33]. In comparison to [43,44], it provides a response faster than the sliding mode controller [44] but slower than the disturbance observer-based controller [44]. However, the steady state error of the proposed controller is smaller than that of [43,44].

In Figure 8 and Figure 9, the angular velocities of the proposed control scheme are within the preset constraint (red lines). While the controllers from [26,33,43,44] result in the angular velocities of two joints exceeding the constraints a $t=0~2 s$.

case 2: 

In the presence of system uncertainty, external disturbance and friction.

In this case, the system parameters used in all the five controllers are different to that in the dynamics model of the manipulator to indicate the parametric uncertainty. Namely, $∆M$, $∆C$ and $∆G$ is taken as 20% of $M$, $C$ and $G$. Moreover, the friction and external disturbance, which are not known by the controllers, are applied on the dynamics model.

The friction model is from [33].

$$\mathit{F}\left(\dot{\mathit{q}}\right)=\left[\begin{array}{c}{f}_{s1}\left[\tanh \left(f_{s2}{\dot{q}}_1\right)-\tan \mathrm{h}\left(f_{s3}{\dot{q}}_1\right)\right]+f_{c1}\tanh \left(f_{c2}{\dot{q}}_1\right)+f_v{\dot{q}}_1\\ f_{s1}\left[\tanh \left(f_{s2}{\dot{q}}_2\right)-\tan \mathrm{h}\left(f_{s3}{\dot{q}}_2\right)\right]+f_{c1}\tanh \left(f_{c2}{\dot{q}}_2\right)+f_v{\dot{q}}_2\end{array}\right]$$

The parameters of the friction model are given as: $f_{s1}=20$, $f_{s2}=5$, $f_{s3}=3$, $f_{c1}=10$, $f_{c2}=2$ and $f_v=10$.

It is widely seen to use triangular functions as the unknown disturbance in the literature of the robotic system control, such as [13,36,45,46]. Therefore, in this paper, the external disturbance is in the form of the triangular functions.

$${\mathit{\tau }}_{\mathit{d}}=\left[\begin{array}{c}0.78\sin \left(\frac{\pi }{3}t+\frac{\pi }{4}\right)+0.065\sin \left(\frac{\pi }{10}t+\frac{\pi }{4}\right)\\ 0.58\cos \left(\frac{\pi }{3}t+\frac{\pi }{4}\right)+0.091\sin \left(\frac{\pi }{10}t+\frac{\pi }{4}\right)\end{array}\right]$$

The comparisons of tracking performance and tracking error in the presence of system parametric uncertainty, friction and external disturbance are shown in Figure 10, Figure 11, Figure 12 and Figure 13, respectively. Clearly, the proposed controller can achieve the smallest steady state errors, which means the robustness to the lumped uncertainty and unknown disturbance. The converging rate of tracking errors of the proposed controller is faster than the controller [33,43] but slower than the controller [26,44]. It is because the preset constraint of angular velocity (black lines in Figure 14 and Figure 15) limits the converging rate of tracking errors. Therefore, the converging rate of tracking errors could be increased by increasing the value of velocity constraint (${\mathsf{\Lambda }}_i$) (e.g., applying some better driving motors that have a greater maximum rotational speed to drive the joints of manipulator).

The computed control torques are shown in Figure 16 and Figure 17. The chattering effect of the proposed controller occurs at the initial stage because of the two following factors: 1. the increasing value of switching gain ${\widehat{d}}_i$ to handle disturbance at the initial stage, 2. the fuzzy Q learning mechanism tried some bad action candidates that lead to an undesirable consequence. After the initial stage ($t>8 \mathrm{s}$), it is observed that the proposed controller shows the smoothest control torque compared to [26,33,43,44]. Figure 18 shows the values of switching gain ${\widehat{d}}_1$ and ${\widehat{d}}_2$, it is clear the ${\widehat{d}}_1$ and ${\widehat{d}}_2$ will decrease to a small value to avoid a chattering in steady state regardless of the disturbance and uncertainty.

Notably, in both case 1 and case 2, the proposed controller can make the angular positions and angular velocities of two joints to be within their constraints during the whole period of position tracking. More precisely, the angular position of each joint is always between the two black dash lines in Figure 6, Figure 7, Figure 10 and Figure 11. Meanwhile, the angular velocity of each joint is always between the two black dash lines in Figure 8, Figure 9, Figure 14 and Figure 15.

5. Conclusions

This paper proposed a novel adaptive control scheme utilizing TDE and RL for the angular position tracking control of robotic manipulators. The proposed control scheme can achieve a good tracking accuracy and a fast tracking performance even when subject to the system uncertainty and unknown disturbance. Moreover, the angular position and angular velocity of each joint of the manipulator are guaranteed to be within their preset constraints. The boundness of tracking errors and the stability of the robotic system controlled by the proposed controller are proven by Lyapunov theory. Notably, the stability will not be breached by the RL trying some bad action candidates, which ensures a safe environment for RL to explore the optimal policy. Simulation results validate the effectiveness of the proposed control scheme.