Multi-Phase Focused PID Adaptive Tuning with Reinforcement Learning

Abstract

The Proportional-Integral-Derivative (PID) controller, a fundamental element in industrial control systems, plays a pivotal role in regulating an extensive array of controlled objects. Accurate and rapid adaptive tuning of PID controllers holds significant practical value in fields such as mechatronics, robotics, and automatic control. The three parameters of the PID controller exert a substantial influence on control performance, rendering the tuning of these parameters an area of significant interest within related research fields. Numerous tuning techniques are widely employed to optimize its functionality. Nonetheless, their adaptability and control stability may be constrained in situations where prior knowledge is inadequate. In this paper, a multi-phase focused PID adaptive tuning method is introduced, leveraging the deep deterministic policy gradient (DDPG) algorithm to automatically establish reference values for PID tuning. This method constrains agent actions in multiple phases based on the reward thresholds, allowing the output PID parameters to focus within the stable region, which provides enhanced adaptability and maintains the stability of the PID controller even with limited prior knowledge. To counteract the potential issue of a vanishing gradient following action constraints, a residual structure is incorporated into the actor network. The results of experiments conducted on both first-order and second-order systems demonstrate that the proposed method can reduce the tracking error of a PID controller by 16–30% compared with the baseline methods without a loss in stability.

1. Introduction

In this section, the characteristics and control methods of PID controllers are introduced, leading to the introduction of the methods for controlling PID controllers using reinforcement learning. Related works are also discussed here. Finally, the main contributions are summarized in light of the above content.

1.1. Motivation

PID controllers are widely used in industrial process control, with over 90% of applications relying on this simple yet highly effective controller design [1,2]. PID controllers are typically designed by tuning the parameters $K_p$, $K_i$, and $K_d$ [3]. However, as linear controllers, PID controllers have limitations in achieving sophisticated control [4] because of parameter tuning often relying on the guidance of empirical data [5], which makes accurate PID parameter tuning difficult for complex systems. Developing adaptive parameter tuning PID controllers that can adapt to multiple task scenarios has become one of the key research topics in the field of control [6,7].

PID parameters can be optimized using two main methods: classical tuning methods and optimization methods [8,9]. Classical methods often rely on heuristic tuning or model construction to address specific engineering challenges [10]. However, these methods, such as the Ziegler–Nichols method [11] and the Cohen–Coon method [12], make assumptions about the underlying model or attempt to estimate the best approximation of the PID parameters [13]. Therefore, small deviations from the model assumptions may result in significant discrepancies between the predicted and actual system behavior. Optimization methods can obtain optimal PID parameters by precisely modeling the problem. Studies [14,15,16] show that optimization methods can obtain optimal PID parameters by accurately modeling the problem. Nevertheless, an accurate model is often difficult to obtain in practical engineering problems. As a consequence, developing PID adaptive tuning methods that can adapt to a wide range of task scenarios remains a major research challenge in the control field.

Neural network techniques, which provide a high-dimensional mapping relationship between inputs and outputs as supervised learning issues, are important techniques for PID adaptive tuning. Neural networks have been shown to outperform some other intelligent methods in terms of PID adaptive tuning [17,18]. However, again, collecting the right data labels can be challenging in actual engineering problems [19].

To address the limitations in supervised learning, this paper attempts to apply reinforcement learning (RL)—a model-free method—to PID adaptive tuning, as it allows agents to identify optimal behavior through trial-and-error interactions with the environment [20,21]. Unlike neural network methods, RL methods do not require labeled data; instead, agents interact with environments and modify their exploration policies through reward functions to find the optimal policy. Our goal is to utilize an agent that is allowed to explore freely in the environment to determine the optimal PID parameters without the use of an engineering model. However, the instability and windup phenomena [22] caused by the use of a truly random exploration are obviously unacceptable. Therefore, many studies guide the exploration of the agent by pre-setting empirical PID parameters based on engineering experience as reference values. In this work, the aim is to automatically establish and modify PID reference values throughout the execution process to ensure stability in the absence of prior knowledge. PID parameter tuning is characterized by its continuous correspondence with tracking performance [11] within the boundary conditions, where a small change in parameters does not cause a significant change in control effect. This feature provides guidance for balancing exploration and exploitation when using RL to solve the problem of PID adaptive tuning. When the PID output reaches a certain level of control, the agent’s policy is constrained and guided, which helps to further improve the performance of the PID controller without causing output oscillation.

1.2. Related Works

Recently, RL-based methods have been widely applied in the field of PID parameter tuning. The authors of [23] proposed the application of RL to PID controller design to improve the performance of agents in OpenAI, and the average total number of control steps per episode from episode 5000 to episode 6000 was reduced by more than 5%. The authors of [24] developed an RL-based PID tuning method called continuous actor–critic learning automata (CACLA) for human-in-the-loop physical assistive control. In another study [25], the authors addressed the problem of simultaneously outputting multiple parameters in PID controllers based on an RL framework. Ref. [26] proposed an expert agent-based system based on RL for self-adapting multiple low-level PID controllers in mobile robots.

Nevertheless, in the above-cited works, the agent’s policy is more focused on exploration in the early phases of iteration, resulting in excessive PID output amplitude and integral saturation, which can destabilize the system. Aiming at the closed-loop stability of RL-based PID tuning methods, the authors of [27] expressed a PID controller as a shallow neural network of the actor network, and improved the stability of the PID controller. The results show that the moving average of the number of time steps per episode before the PI controller was reduced by nearly 50%. Ref. [10] proposed an RL-based stability-preserving PID adaptive tuning framework that guarantees the controller’s stability in the entire closed-loop process and found that this approach reduced the MSE tracking error by more than 10% compared with other methods. However, this method requires a known set of PID parameters as reference values to guide the agent’s exploration. Ref. [19] initialized all the PID parameters to 0 without prior knowledge based on simultaneous calculation of the policy and value function using the RBF network, but this method lacks universality as the network topology is tailored to a specific system.

Based on the analysis presented above, the aim of our research is to develop an RL-based PID adaptive tuning framework that can be applied to various systems with limited prior knowledge while ensuring PID controller stability and improved tracking performance. To mitigate the risk of instability, this paper intends to regulate the action of the agent by constraining its exploration, drawing inspiration from the theory of constraint reinforcement learning [28]. This is achieved by imposing instantaneous constraints on the agent’s actions, a technique for which a theoretical framework has been proposed in the literature [29]. By incorporating explicitly constrained or softly penalized constraint violations [30], the agent can achieve the desired result while avoiding unstructured exploration.

1.3. Main Contributions

In this work, a multi-phase focused deep deterministic policy gradient (MF-DDPG) framework for PID adaptive tuning was proposed. The method ensures the stability of the PID controller in the closed-loop system under limited prior knowledge conditions by focusing on the agent actions in phases, while achieving fast convergence performance and better control effect. The main contributions of our work can be summarized as:

• To balance exploration and exploitation in RL-based PID adaptive tuning, a multi-phase focused PID parameter adaptive tuning framework was proposed based on RL. To ensure the agent’s output PID parameters remain within the stable region, the agent’s action exploration is constrained to remain close to a corresponding reference value of the PID parameters during each phase, a reference value that is then adjusted when the agent performance satisfies a preset reward threshold condition. In this framework, the PID parameter search space is continuously refined from coarse to fine across the training phases, leading to a near-monotonic improvement in the PID parameters’ performance.
• To solve the problem of obtaining a reference value for PID adaptive tuning under the constraint of limited prior knowledge, a mechanism to automatically determine a reference value of the PID parameters was introduced in the multi-phase focusing process. At each phase, the agent automatically establishes a reference value of the PID parameters based on the current phase’s reward threshold and explores within a certain range around this reference value. The reference values established in different phases provide a baseline for the tracking performance of the PID controller, ensuring that the PID controller continuously focuses on improving control performance. The proposed method achieves impressive control performance without prior knowledge of the precise reference values of the PID parameters.
• To address the challenge of a vanishing gradient, which can arise when constraining the output of the actor net of the proposed algorithm, a residual structure was added to the actor net. After applying multi-phase action constraints, the numerical space of the output for the actor net in RL is compressed, leading to a small range of output changes that may cause the gradient to vanish during back-propagation, hindering agent exploration. To overcome this issue, a residual structure was introduced from the shallow layer to the deeper layer of the actor net, preventing the gradient from vanishing and enabling the agent to maintain a certain level of exploration even after multiple action constraints.

1.4. Paper Organization

The outline of the paper is as follows. Section 2 describes the preliminary stages of our work. Section 3 is the description of the proposed MF-DDPG framework for adaptive PID tuning, and the stability maintaining mechanism of the framework. Section 4 provides the experimental results and analysis of the method. Section 5 is the conclusion of the work.

2. Preliminary Work

In this section, the principles and structure of PID controllers are introduced, as well as the foundational knowledge related to reinforcement learning algorithms. Building upon this foundation, the multi-phase focused framework is introduced and a detailed analysis is provided of its principles and workflow.

2.1. PID Control

PID controllers can be divided into two main types: series PID and parallel PID controllers [11]. In series PID controllers, the parameters are coupled, whereas in parallel PID controllers, the three PID parameters are decoupled. The latter approach offers easier tuning and wider applicability. Therefore, this paper primarily focuses on parallel PID controllers:

$$u\left(t\right)=K_{p}e\left(t\right)+K_i{\int }_0^{t}e\left(t\right)+K_d\frac{\mathrm{d}e\left(t\right)}{\mathrm{d}t}$$

(1)

where $e\left(t\right)=s\left(t\right)-y\left(t\right)$ is the difference between the setpoint $s\left(t\right)$ and the output $y\left(t\right)$. Equation (1) represents the PID control system in continuous form; however, in many applications, PID controllers are stated in terms of discrete time conditions, as demonstrated in (2):

$$u\left(t_n\right)=K_{p}e\left(t_n\right)+K_{i}I\left(t_n\right)+K_{d}D\left(t_n\right)$$

(2)

where $I\left(t_n\right)={\sum }_{i=1}^{n}e\left(t_i\right)△t$ and $D\left(t_n\right)=\frac{e\left(t_n\right)-e\left(t_{n-1}\right)}{△t}$ are the integrated and differentiated errors of a discrete PID controller, respectively, in which $△t>0$. This paper focuses on investigating PID controllers based on Equation (2) and more specifically, how to address the integral saturation issue during the adjustment of PID controllers. Actuator saturation is a common problem that can cause instability in PID controllers. When a PID controller saturates, it can become an open-loop system [27], causing system errors to accumulate and leading to a loss of stability known as “windup”. To mitigate the effects of windup, many studies have employed anti-windup techniques [22]. These techniques involve setting upper and lower bounds for the controller’s output to prevent the accumulation of integral terms by accounting for saturation, as shown below:

$$sat\left(u\right)=\left\{\begin{array}{c}{u}_{min},ifu<u_{min}\hfill \\ u,if{u}_{min}<u<u_{max}\hfill \\ u_{max},ifu>u_{max}\hfill \end{array}\right..$$

(3)

According to (3), when the output is lower than the lower bound, the controller will take $u_{min}$ as the output, and when the output is higher than the upper bound, it will take $u_{max}$ as the output. This may prevent a build up in the integrator incremental error accumulation [27]. Naturally, this is a fundamental technique for preventing integral saturation, and this paper will further develop it, as will be covered in more depth later.

2.2. Deep Deterministic Policy Gradient Algorithm

The Markov Decision Process (MDP) is the foundation of RL [31], which defines a mathematical model of sequential decision-making where the current state depends only on the preceding state and action [32]. The MDP definition is satisfied for PID tuning as the PID output error from the previous step is the input for the next step. RL consists of four components: state $s_k\in \mathcal{S}$, action $a_k\in \mathcal{A}$, policy, and reward $\mathcal{R}$. PID parameter tuning is equivalent to an agent’s random exploration of the environment, where various reward functions correspond to the outputs of different parameters. The policy maps the state $s_k$ and action $a_k$ to an action probability distribution $\pi \left(a_k\right|s_k)$, while the agent receives a reward $r_k$ for the action $a_k$ it took in the current state $s_k$. The transition from one state to the next state $s_{t+1}$ occurs with a certain transition probability $p\left(s_{t+1}\right|s_t,a_t)$ based on the action $a_t$ taken in the current state $s_t$ (Figure 1). Here, t represents the time step during an episode with a duration of T, where the agent takes one step at each time step. The $s_t$, $a_t$, and $r_t$ corresponds to the state, action, and reward values, respectively, resulting from the interaction between the agent and the environment at time t. The goal of RL is to maximize the return by identifying the optimal parameters.

In the context of RL, a policy can be categorized into two forms: deterministic or stochastic. For the PID parameter tuning problem, the parameters output by the agent are in continuous vector form, making it more appropriate to use a deterministic policy [33]. Therefore, in the current work, the deep deterministic policy gradient (DDPG) algorithm was used as the foundation of the research framework. The paper defines the cumulative reward over a policy trajectory with a finite number of steps, and aims to maximize the expected cumulative discounted reward starting from a certain state:

$$J\left(\pi \right)={\mathbb{E}}_{\tau \sim \pi }\left(\sum _{t=0}^T{\gamma }^t{r}_t\right)$$

(4)

where $\tau$ represents the trajectory of the sampled policy, ${\sum }_{t=0}^T{\gamma }^t{r}_t$ is the discounted return starting at $t=0$. In order to maximize $J\left(\pi \right)$, the policy is parameterized with $\theta$ and solved by a neural network. Equation (4) can be rewritten as

$$J\left({\pi }_{\theta }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left(\sum _{t=0}^T{\gamma }^t{r}_t\right).$$

(5)

Through gradient ascent, the optimization objective becomes

$$\theta \leftarrow \theta +\alpha {▽}_{\theta }J\left({\pi }_{\theta }\right)$$

(6)

where $\alpha$ is the learning rate. A policy-based method provides a more direct solution than a value-based method. However, due to the strong randomness of the policy and significant variance in the network updating process, it can be difficult to obtain the best solution. To estimate the action value function, a critic network is used in a value-based optimization network. With parameterized value function approximation, the actor parameter $\theta$ is used for the policy network, while the critic parameter $\omega$ is used for the value network, forming an Actor–Critic (AC) structure. According to the policy gradient theorem [34], the policy parameters can be updated by following the gradient of the expected reward with respect to the policy parameters

$${▽}_{\theta }J\left({\pi }_{\theta }\right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\sum _{t=0}^T{▽}_{\theta }{log}_{}{\pi }_{\theta }\left(a_t\right|s_t)Q^{\omega }(a_t,s_t)$$

(7)

where $Q^{\omega }(a_t,s_t)$ represents the value function of action $a_t$ in state $s_t$ [31,35]. For a deterministic policy given as ${\mu }_{\omega }:\mathcal{S}\to \mathcal{A}$, every action value is computed by

$$a_t=\mu \left(s_t\right|{\theta }_t)+{\mathcal{N}}_t$$

(8)

without sampling from stochastic policies, where ${\mathcal{N}}_t$ is stochastic noise introduced to encourage exploration. The DDPG algorithm is commonly used to facilitate agent exploration in continuous action spaces. The algorithm establishes Q-value nets $Q(s,a|\omega )$ and corresponding target nets $Q^{{}^{\prime }}(s,a|{\omega }^{{}^{\prime }})$ for the critic network, as well as policy nets $\mu \left(s\right|\theta )$ and corresponding target nets ${\mu }^{{}^{\prime }}\left(s\right|{\theta }^{{}^{\prime }})$ for the actor network. To reduce correlations between samples, a replay buffer $\mathcal{D}$ is utilized to store previously sampled data $(s_i,a_i,r_i,s_{i+1})$. For updating the networks’ parameters, a random batch N of transition experiences is selected from $\mathcal{D}$. Based on the method outlined in [33], the critic network is updated by minimizing the MSE loss of

$$L=\frac{1}{N}\sum _i^{}{(y_i-Q(s_i,a_i|\omega ))}^2$$

(9)

where $y_i=r_i+\gamma Q^{{}^{\prime }}(s_{i+1},{\mu }^{{}^{\prime }}\left(s_{i+1}\right|{\theta }^{{}^{\prime }})|{\omega }^{{}^{\prime }})$ is computed by target nets. The actor network is updated by

$${▽}_{\theta }J\approx \frac{1}{N}\sum _i^{}{▽}_{a}Q(s,a|\omega ){{|}_{s=s_i,a=\mu \left(s_i\right)}{▽}_{\theta }\mu \left(s\right|\theta )|}_{s_i}.$$

(10)

The networks iteratively update the parameters according to (9) and (10) until convergence.

3. Multi-Phase Focused PID Adaptive Tuning with DDPG

In this section, the methodology for utilizing MF-DDPG to automatically update the parameters of PID controllers is introduced and the proposed method for PID adaptive tuning using RL is described. Figure 2 illustrates the framework of the proposed method, showing the coupling mechanism between the PID controller and the multi-phase focused RL algorithm. The proposed method is based on multi-phase action constraints that restrict the agents’ exploration to a maximum within the stable region by imposing constraints and gradually focusing actions. This method requires only approximate knowledge of the PID parameters’ range, without the need to establish a reference value for them. The reference value is automatically determined along with the switching of the corresponding phases.

• Environment and agent: The PID closed-loop control operation serves as the environment for the agent, and each step where the PID controller completes a full closed-loop control is an interaction between the agent and the environment. The resulting experience data $<s_t,a_t,r_t,s_{t+1}>$ are stored in the replay buffer. By iteratively learning the agent’s initial random exploration policy, RL aims to eventually learn the environment’s distribution and guide the agent’s exploration towards the optimal policy. To update the critic and actor networks, a set of experience data $<s_i,a_i,r_i,s_{i+1}>$ are randomly selected from the replay buffer every K iterations. The updated actor network then computes the next action $a_{t+1}$ based on the current state $s_t$ of the agent, which corresponds to the parameters of the subsequent PID closed-loop operation.
• State and action: The state represents the information obtained by the interaction between the agent and the environment. The agent obtains the reward $r_t$ by evaluating the current state $s_t$. In the PID control problem, many studies [36,37] take the entire episodic trajectories of inputs and outputs of t-th closed-loop operation

  $$y_t={[y_0^t,y_1^t,y_2^t,\dots ,y_L^t]}^T,u_t={[u_0^t,u_1^t,u_2^t,\dots ,u_L^t]}^T$$

  (11)

  and

  $$s_t={[y_t,u_t]}^T$$

  (12)

  as the state, where L represents the maximum number of time steps of the episodic trajectories. Yet this will introduce the problem of overly high-dimensional inputs to the RL networks. In another study [10], episodic data are described as a state customized to equal the time step, reducing the input dimension, but still potentially resulting in information redundancy. Our observations show that the early stages of the control process exhibit significant output fluctuations, while the output value stabilizes progressively towards the setpoint in the later stages, known as the tuning characteristic of PID control. To capture the impact of the episode’s setting, the time frame with the most pronounced output fluctuations was selected, representing the state of each episode with the first H steps of the closed-loop process. Additionally, in the early stages of exploration, the agent frequently explores the unstable interval of PID parameters, leading to excessive closed-loop operation amplitude and output oscillation, making the output oscillation in the first H steps more noticeable. To address this issue, the system’s output from the previous H steps ${[y_0^t,y_1^t,y_2^t,\dots ,y_H^t]}^T$ was normalized and used as the state representation. This simplifies the agent state and reduces network dimension while preserving the observed environmental information. In each episode, the action $a_t$ is the episodic PID parameters $a_t={[K_p^t,K_i^t,K_d^t]}^T$, and action $a_t$ is used to guide the episodic PID closed-loop operation.
• Reward: To ensure effective control, the reward function of the PID controller should be designed to have a positive correlation with its performance. In particular, the amplitude of the system output $y\left(t\right)$ should be minimized, as illustrated in Figure 3, and the output should converge to the setpoint rapidly. Accordingly, our reward function incorporates both the time steps and the error $e\left(t\right)$ of the complete closed-loop trajectories. The reward function is described as

  $$r_t=-{\left(\frac{{\sum }_0^N\left|e_n^t\right|}{{\lambda }_1}+{\lambda }_{2}N\right)}^2$$

  (13)

  where $e_n^t=y_n^t-s\left(t\right),n=0,1,2,3,\dots ,N$, represents the n-th step of the t-th episodic closed-loop operation. The weights of the error and time step in the reward function are denoted by ${\lambda }_1$ and ${\lambda }_2$, respectively. By adjusting these weights, this paper can comprehensively consider trajectory-based metrics of both the error $e\left(t\right)$ and the number of time steps N during the PID tuning process. This enables us to utilize a trade-off between output amplitude and time steps in reward design. A smaller amplitude of the output $y\left(t\right)$ corresponds to a shorter time interval and a higher reward. It is worth noting that all rewards $r\left(t\right)$ are less than 0.

The proposed method utilizes the anti-windup method described in (3) to establish an initial upper and lower bound for the vectors of $O_{\max }$ and $O_{\min }$, respectively, for the PID parameters. However, the output of the activation layer of the actor net does not necessarily fall within the range of PID parameters, and directly constraining the actions through normalization [38] compresses the action space and restricts the agent from fully exploring the environment. Therefore, it is necessary to establish a comprehensive mapping between the action interval and $[O_{\min },O_{\max }]$. We define

$$a_t=\frac{O_{max}-O_{min}}{\mu {\left(s\right|\theta )}_{max}-\mu {\left(s\right|\theta )}_{min}}\odot \left(\mu \left(s_t\right|\theta )+{\mathcal{N}}_t-\mu {\left(s\right|\theta )}_{min}\right)+O_{min}$$

(14)

were O-U noise [35] is used as ${\mathcal{N}}_t$. $\mu {\left(s\right|\theta )}_{max}$ and $\mu {\left(s\right|\theta )}_{min}$ are the upper and lowed bounds of the activation function, respectively. To ensure comprehensive exploration by the agent and guarantee stability of the closed-loop process, a linear mapping relationship between the activation function output interval and the PID parameter interval can be established. This linear relationship, represented by Equation (14), can be viewed as adding an analogous linear activation layer to the actor network, thus leaving the calculation in DDPG unaffected. This is regarded as the first phase of agent exploration. However, this mapping alone may not ensure stability, and thus, the paper further constrains actions based on various thresholds to ensure stability of PID adaptive tuning with RL. These thresholds are established based on the reward function, and in this study, two thresholds, $R^{th1}$ and $R^{th2}$, are set with $R^{th1}$ being smaller than $R^{th2}$, and the reference values for the PID controller are automatically established each time the reward satisfies the criteria, allowing the agent to continue exploring within a specific range of the reference values until the algorithm converges. The reference value of the PID parameters is initialized as

$$\mathcal{B}={[0,0,0]}^T.$$

(15)

At the beginning of the algorithm, the agent takes actions according to (14) and obtains the episodic tuple of experience $<s_t,a_t,r_t,s_{t+1}>$. Once at $t_i$, if this satisfies $r_{t_i}\ge R^{th1}$, then the reference value is set as

$$\mathcal{B}=a_{t_i}={[K_p^{t_i},K_i^{t_i},K_d^{t_i}]}^T.$$

(16)

Once the new reference value of the PID parameters has been established, the action constraints are switched to initiate the second phase of exploration. For $K_p^{t_i}$, $K_i^{t_i}$ and $K_d^{t_i}$, the corresponding exploration ranges ${\psi }_p^1$, ${\psi }_i^1$ and ${\psi }_d^1$ are set, respectively. The range of PID parameters in the second phase is

$${[K_p^t,K_i^t,K_d^t]}^T\in [({\mathbf{1}}_3-\mathsf{\Psi })\odot \mathcal{B},({\mathbf{1}}_3+\mathsf{\Psi })\odot \mathcal{B}]$$

(17)

where $\mathsf{\Psi }={\mathsf{\Psi }}^1={[{\psi }_p^1,{\psi }_i^1,{\psi }_d^1]}^T$. Then the new mapping of the action to the PID parameters is

$$a_t=\frac{2\mathsf{\Psi }\odot \mathcal{B}}{\mu {\left(s\right|\theta )}_{max}-\mu {\left(s\right|\theta )}_{min}}\odot \left(\mu \left(s_t\right|\theta )+{\mathcal{N}}_t-\mu {\left(s\right|\theta )}_{min}\right)+\left({\mathbf{1}}_3-\mathsf{\Psi }\right)\odot \mathcal{B}.$$

(18)

Equation (18) indicates that the agent explores within a certain range of the current reference value which guarantees a baseline performance and further ensures the stability of PID closed-loop control based on the first phase. Similar to (14), the $a_t$ in (18) also satisfies a linear relationship with the $\mu \left(s_t\right|\theta )$ of the actor network. By setting the threshold $R^{th1}$, the agent is able to explore in the stable region of PID parameters. To obtain better parameters and a more stable closed-loop tuning region, further constraining the agent’s exploration space in the stable region is proposed based on the second phase. If the reward $r_t$ at $t_j$ satisfies $r_{t_j}\ge R^{th2}$, then $\mathcal{B}$ is set as

$$\mathcal{B}=a_{t_j}={[K_p^{t_j},K_i^{t_j},K_d^{t_j}]}^T$$

(19)

and the exploration range is $\mathsf{\Psi }={\mathsf{\Psi }}^2={[{\psi }_p^2,{\psi }_i^2,{\psi }_d^2]}^T$. The constraint on the agent’s action will then repeat (17) and (18) until convergence. The process of constraining the actions in phases is similar to the focusing process of a camera. By focusing on the action through constraints corresponding to $R^{th1}$, the PID parameters are stabilized within the stable region, which corresponds to “coarse focusing”. Further constraints are applied to the action when the threshold of $R^{th2}$ are met, so that the PID parameters are limited to smaller parameter range, achieving a more refined control effect. This process corresponds to “fine focusing”. Figure 4 illustrates the overall flowchart of the algorithm, and Algorithm 1 provides the pseudo-code of the proposed algorithm.

To reflect the proposed approach, corresponding modifications to the actor net and critic net in the DDPG framwork were made. Before the action switching phases of exploration, the agent may enter the unstable region of the PID, resulting in a large output amplitude that could cause the agent to fall into the saturated region of the $tanh$ activation function [39], as previously explained. To mitigate this issue during computation, a normalization layer [40] was added before each layer of the network. Based on the principle of gradient back-propagation [41] in neural networks, after focusing actions through multiple phases, the output of the neural network changes very little, and, as a result, the gradient update of the hidden layer can become very slow, particularly for shallow neural networks, where the gradient can vanish entirely. To address this issue, a residual structure was added from the first fully connected layer to the third fully connected layer for the actor net (the actor net has a total of three fully connected layers). The network structure is depicted in Figure 5. To represent the mapping relationship of two layers in a network, denoted by $\mathcal{F}\left(X\right)$ where X is the output of the shallow network, a basic shortcut was incorporated as a residual connection [42]. This results in the mapping relationship becoming $\mathcal{F}\left(X\right)+X$, and during gradient back-propagation, the partial derivative of $\mathcal{F}\left(X\right)+X$ with respect to the shallow network parameters will not be 0, thus avoiding the issue of gradients vanishing.

Remark 1.

The proposed method circumvents the need for direct action constraints and instead regulates the threshold setting through the reward function. Directly constraining the trajectory of a closed-loop process to ensure exploration in the stable region is challenging due to the lack of prior knowledge. The reward function, which is positively correlated with the stability of PID adaptive tuning and incorporates both amplitude and time considerations, is used to adjust the threshold value. As a result, this method allows for flexible and adaptable threshold setting based on the reward function.

Algorithm 1 Multi-phase Focused DDPG-based PID Adaptive Tuning Algorithm

Input:  Actor network parameters $\theta$, critic network parameters $\omega$, and the value range of PID parameters $[O_{\min },O_{\max }]$; Output:  Action $a_t$ of the agent, optimized $\theta$ and $\omega$; 1: Initialize target nets ${\mu }^{{}^{\prime }}$ and $Q^{{}^{\prime }}$ with weights ${\theta }^{{}^{\prime }}\leftarrow \theta$, ${\omega }^{{}^{\prime }}\leftarrow \omega$, PID reference value $\mathcal{B}=[0,0,0]$, exploration range ${\mathsf{\Psi }}^1={[{\psi }_p^1,{\psi }_i^1,{\psi }_d^1]}^T$, ${\mathsf{\Psi }}^2={[{\psi }_p^2,{\psi }_i^2,{\psi }_d^2]}^T$, replay buffer $\mathcal{D}$, Switch = False; 2: for t = 0, 1, 2, …, T do 3:  Reset the environment and receive initial observation state $s_0$; 4:  if Switch == False then 5:   Compute $a_t$ as given in (14); 6:  else 7:   Compute $a_t$ as given in (18); 8:  end if 9:  Perform a complete PID closed-loop control process with parameter $a_t$, and receive the next state $s_{t+1}$, reward $r_t$; 10:  if $r_t\ge R^{th1}$ then 11:   Switch = True, $\mathcal{B}=a_t$ and stay still until $r_t\ge R^{th2}$, $\mathsf{\Psi }={\mathsf{\Psi }}^1$; 12:   if $r_t\ge R^{th2}$ then 13:    $\mathcal{B}=a_t$ and remain unchanged, $\mathsf{\Psi }={\mathsf{\Psi }}^2$; 14:   end if 15:  end if 16:  Store transition $(s_t,a_t,r_t,s_{t+1})$ in buffer $\mathcal{D}$; 17:  Sample a batch of N transitions $(s_i,a_i,r_i,s_{i+1})$ from buffer $\mathcal{D}$; 18:  Compute the target Q value of each sampled transition: $y_i=r_i+\gamma Q^{{}^{\prime }}(s_{i+1},{\mu }^{{}^{\prime }}\left(s_{i+1}\right|{\theta }^{{}^{\prime }})|{\omega }^{{}^{\prime }})$; 19:  Update critic network by minimizing the loss as given in (19); 20:  Update actor network by one-step sampled gradient descent as given in (20); 21:  Update target networks for every K steps ($0<\tau <1$): $${\theta }^{{}^{\prime }}⟵\tau \theta +(1-\tau ){\theta }^{{}^{\prime }},{\omega }^{{}^{\prime }}⟵\tau \omega +(1-\tau ){\omega }^{{}^{\prime }}$$ 22:  Set $s_t=s_{t+1}$; 23: end for

4. Experimental Results and Analysis

In this section, experimental validations of the proposed algorithm using both first-order and second-order systems are conducted. The analysis of the experimental results demonstrates that the algorithm adequately achieves adaptive tuning of PID controllers and significantly improves their performance.

4.1. Experiment Settings

In this section, the validation of the proposed method is presented using a first-order system and a second-order system. The first-order system is modeled by

$$G\left(s\right)=\frac{0.5}{s+1}$$

(20)

and the second-order system is modeled by

$$G\left(s\right)=\frac{1}{s^2+10s+20}$$

(21)

with initial condition $y\left(0\right)=0$. The RL algorithm parameters used in the experiments are presented in

Table 1. Parameters used for DDPG algorithm.

Parameters	Actor Net	Critic Net
Structure for networks	[32, 16, 32, 1]	[32, 16, 1]
Learning rate	0.001	0.001
Activation function	tanh	relu
Replacement factor $\tau$	0.01	0.01
Optimization function	Adam	Adam
Memory size for replay buffer	2000
Batch size	32
Discount factor $\gamma$	0.9
Training length	3000
State dimension H	5

. To ensure a fair comparison, the PID parameter ranges for the first-order system and the second-order system were set to $K_p\in [0,2],K_i\in [0,1.5],K_d\in [0,0.15]$ and $K_p\in [0,100],K_i\in [0,100],K_d\in [0,30]$, respectively, based on their tuning characteristics. The algorithm parameters were initialized based on the PID range, and the PID parameters interval was kept within the predetermined range.

Table 2. Parameters of multi-phase focused PID adaptive tuning algorithm.

Parameters	Values (1st) 1	Values (2nd) 1
Reward weights ${\lambda }_1$	10	200
Reward weights ${\lambda }_2$	0.1	0.001
Reward threshold $R^{th1}$	−10	−10
Reward threshold $R^{th2}$	−5	−5
Exploration range ${\mathsf{\Psi }}^1$	[0.15, 0.15, 0.2]	[0.15, 0.15, 0.2]
Exploration range ${\mathsf{\Psi }}^2$	[0.15, 0.15, 0.2]	[0.15, 0.15, 0.2]
Episodic length	20	10

¹ 1st represents the first-order system and 2nd represents the second-order system.

shows the proposed algorithm’s parameters. The sample time for the first-order system and the second-order system were set to 1 s and 0.01 s, respectively. At each sample, the system performs a closed-loop operation to determine the error $e\left(t\right)$ and system output $y\left(t\right)$. The second-order system has 50 times more closed-loop data in each episode than the first-order system due to the different sampling rates. To facilitate further analysis, the reward weights of the two systems, ${\lambda }_1$ and ${\lambda }_2$, were adjusted so that the rewards were of the same order of magnitude. The paper also set the same exploration interval for ${\mathsf{\Psi }}^1$ and ${\mathsf{\Psi }}^2$ since the first-order and second-order systems were not sensitive to differential setting of the exploration range before and after the action range switching. Moreover, the exploration rate for $K_d$ was increased significantly over the rates for $K_p$ and $K_i$ to facilitate the experiment, as $K_d$ has a smaller range than $K_p$ and $K_i$. Further parameter settings were made based on the PID tuning experience model and the benchmark of the DDPG model.

4.2. Threshold Selection Experiment

In this section, the impact on the proposed method of changing the thresholds is explored. The proposed method uses $R^{th1}$ and $R^{th2}$ as the threshold values to switch the PID reference values when the running reward surpasses these thresholds. Therefore, how these thresholds are set can significantly impact the algorithm’s performance. When action constraint is not employed, the rewards of both systems converge to around −10, as confirmed in the main experiment. To guarantee the algorithm’s fundamental performance, $R^{th1}$ = −10 was set, while $R^{th2}$ should be within the range $(-10,0)$. To determine the optimal threshold value, the threshold between −7 and −3 was set to $R^{th2}$ in the experiments. Figure 6 displays the outcomes. The top row shows the reward curves for various $R^{th2}$ values across the complete episodes, where “benchmark” represents the reward = −5. When $R^{th2}$ is set to −5, the reward function in the first-order system converges to −5, indicating that the algorithm has switched constraints during the three phases, and the PID output remains within a stable range. However, when $R^{th2}$ = −7 or −6, the agent’s exploration is limited, resulting in the reward converging around the corresponding threshold. Conversely, the running reward cannot meet the conditions $r_{t_i}\ge R^{th2}$ when $R^{th2}$ = −4 or −3, causing the reward to stagnate at a lower level. This issue is further highlighted in the optimal PID tracking curve in the lower left plot for a first-order system with various $R^{th2}$ and is also shown in Figure 6’s upper and lower right plots for second-order systems. In the experiments, the threshold selection step size was set to 1. This choice was made because the reward function’s design ensured that variations in the reward value within a range of 1 or less would not have a significant impact on the final PID control performance. Therefore, the step size for threshold design in this experiment is deemed reasonable. Thus, setting $R^{th2}$ to −5 is the optimal choice.

4.3. Experimental Analysis

In this section, we present a comparative analysis of the performance of the MF-DDPG PID adaptive tuning method against existing methods. Figure 7 shows a comparison between the proposed method and the DDPG-based PID tuning method, which serves as the baseline in this study. The top row shows the learning curves of the two methods in both the first-order and second-order systems, with the average reward of our proposed method plotted in red and that of the DDPG method in green. In the absence of action constraint, the learning curves of both systems converge to around −10. However, the proposed method achieves a higher average reward in both systems, indicating the superior stability and performance of the proposed method. Notably, the reward curve of MF-DDPG displays a notable rise when the action constraint is satisfied, which validates the capability of the proposed method. The PID parameter ranges of both methods were also examined over the entire episode to further demonstrate the efficacy of the proposed method. The bottom row of Figure 7 illustrates the distribution range of the parameter $K_p$ in the two systems, ranging from 25% to 75%. The parameter exploration interval of MF-DDPG is more concentrated, with less variance than that of the baseline method. This implies that the action value changes do not disrupt the exploration process but instead lead to the agent’s actions being more focused on the stable region of the PID parameters, thereby enhancing the stability of the closed-loop system. Furthermore, the PID tracking performance of various methods was explored. Of the numerous PID tuning methods available in the literature, the DDPG-based method and the Ziegler–Nichols method [11] were chosen for the comparison. The results are presented in Figure 8 and

Table 3. MSE of different methods (In parentheses is the improvement of MF-DDPG over different methods in terms of tracking MSE).

Methods	System
	First-Order System	Second-Order System
MF-DDPG	2.486	3.441
DDPG	3.483 (28.8%)	4.295 (19.9%)
Z-N	3.702 (32.8%)	4.011 (16.0%)

, with the mean square error (MSE) employed to measure the tracking performance of the PID controllers. the proposed method achieves the lowest MSE in both the first-order and second-order systems, with performance improvements of 16–30% compared to the baseline methods. These findings demonstrate the effectiveness of the MF-DDPG PID tuning method and the closed-loop stability it offers, confirming that it can rapidly track the setpoint while maintaining the smallest PID amplitude during the entire closed-loop process.

Additional experiments were conducted to investigate the changes in the three PID parameters during the switching of action constraints. The outcomes are displayed in Figure 9, which illustrates the trajectories of $K_p$, $K_i$, and $K_d$ during the entire process. The three parameters $K_p$, $K_i$, and $K_d$ exhibit significant variations during the initial stages of the algorithm’s execution. This behavior indicates that the agent is in a phase of random exploration. However, once the rewards meet the threshold conditions, the range of variations for these PID parameters stabilizes rapidly, with only minor fluctuations. It can be observed that the PID parameters enter a stable range throughout the whole episode, as $K_p$, $K_i$, and $K_d$ essentially stabilize within a small interval following the action constraint switching. These results demonstrate that the interval of the parameters can be focused after switching with multiple action constraints, regardless of how big or small the initial interval of the three parameters is. This ensures that the agents’ actions are more likely to be exploitative in the following episode, which is also beneficial for the stability of PID tracking.

The performance of the proposed method was evaluated by examining its ability to maintain consistent setpoint tracking when the setpoint is varied, which is a crucial criterion. To verify the PID adaptive tracking capability of MF-DDPG, step signals were applied to the setpoints of the two systems. Figure 10 illustrates that the proposed method is capable of quickly and stably tracking the new setpoints without experiencing significant oscillations. The PID outputs remain consistently within a stable range. Notably, the second-order system’s higher sampling frequency results in a lower PID output delay than that of the first-order system. This experiment demonstrates that our PID parameter tuning method can generate stable tracking effects and has an excellent adaptive tuning capability.

4.4. Ablation Experiment

A residual structure was applied to the actor network to overcome the problem of a vanishing gradient resulting from the reduction in the action space following the switching of action constraints. To validate the effectiveness of the residual structure, the paper monitored the actor network’s gradient updates throughout the entire learning process. Specifically, the paper focused on the first fully-connected ($FC1$) layer and computed the mean episodic value matrix of the absolute gradients for $FC1$ layer across all episodes, as illustrated in Figure 11. The results demonstrate that actor networks with residual structures have higher average gradients compared to those without, since the gradients of actor networks without residual structures become very small after switching actions, resulting in slow parameter updates. Moreover, the compressed output action space of the actor network limits agent exploration, making it less ideal for learning. However, with the residual structure, the gradients of the actor network do not vanish even when the action space is focused by multiple action constraints. As shown in Figure 11, the gradients of the actor network’s shallow network persist with the residual structure, indicating that the reduction in action space does not negatively impact the parameter updates of the actor network. This is advantageous for updating PID output parameters and achieving a more balanced trade-off between exploration and exploitation.

4.5. Comparison with Existing PID Tuning Methods

In this section, a comparative analysis of the proposed PID adaptive tuning algorithm and several mainstream PID control algorithms is carried out, evaluating the performance of different algorithms in PID adaptive tuning. The paper selects the particle swarm optimization (PSO) tuning method [14], internal model control (IMC) tuning method, and the IMC–Maclaurin (IMC-MAC) closed-loop tuning method [43] for comparison with the proposed MF-DDPG tuning method. The PID controller is applied to a first-order system, as described in Equation (20) (similar considerations apply to second-order systems). The setpoint of the PID controller is designed to be variable, and under the control of the different methods, the respective PID control performance is compared. The results are illustrated in Figure 12. Compared to the PID controllers controlled by the other algorithms, the PID controller controlled by MF-DDPG demonstrates rapid and stable tracking of variable setpoints, with minimal oscillations in the tracking curve when setpoints change.

Table 4. MSE of PID tracking curves with different methods.

Methods	Tracking MSE
MF-DDPG	15.56
PSO	19.63
IMC	18.52
IMC-MAC	17.59

records the MSE for different tracking curves. Experimental results indicate that, in comparison to mainstream PID adaptive tuning algorithms, the MF-DDPG algorithm continues to exhibit advantages.

5. Conclusions

A novel RL-based PID adaptive tuning method was proposed that utilizes multi-phase action constraints. Building on the existing PID adaptive tuning method, this work extends the method by independently exploring parameters using an RL-based algorithm to obtain a finer range of PID parameters. By establishing reference values for PID parameters based on reward thresholds and controlling the agent’s exploration process as a constraint condition, the proposed method can determine more optimal PID tuning parameters with limited prior knowledge while ensuring closed-loop stability during the PID parameter discovery phase. To address the issue of a vanishing gradient caused by the switching of action scale and to expand the agent’s action space, this paper redesigns the reward function and state based on existing literature and adds a residual structure to the actor network. The results of experiments conducted on both first-order and second-order systems demonstrate that our proposed method exhibits a substantial improvement in performance when compared to baseline methods.

In the future, the method will be expanded to nonlinear complex systems to acquire appropriate PID parameters while maintaining system stability.