A Novel Minimal-Cost Power Allocation Strategy for Fuel Cell Hybrid Buses Based on Deep Reinforcement Learning Algorithms

Abstract

Energy management strategy (EMS) is critical for improving the economy of hybrid powertrains and the durability of energy sources. In this paper, a novel EMS based on a twin delayed deep deterministic policy gradient algorithm (TD3) is proposed for a fuel cell hybrid electric bus (FCHEB) to optimize the driving cost of the vehicle. First, a TD3-based energy management strategy is established to embed the limits of battery aging and fuel cell power variation into the strategic framework to fully exploit the economic potential of FCHEB. Second, the TD3-based EMS is compared and analyzed with the deep deterministic policy gradient algorithm (DDPG)-based EMS using real-world collected driving conditions as training data. The results show that the TD3-based EMS has 54.69% higher training efficiency, 36.82% higher learning ability, and 2.45% lower overall vehicle operating cost compared to the DDPG-based EMS, validating the effectiveness of the proposed strategy.

1. Introduction

In response to the energy scarcity crisis and climate warming, clean energy represented by hydrogen has received widespread attention [1,2]. The proton exchange membrane fuel cell (PEMFC) is a power generation system that converts the chemical energy in hydrogen into electrical energy [3,4]. Compared with internal combustion engines, proton exchange membrane fuel cells have the advantages of no pollution, high energy density and efficiency, and low noise [2,5]. PEMFCs and power batteries together constitute fuel cell hybrid electric vehicles (FCHEV) [6], which are able to overcome the problem of soft output voltage of PEMFC systems and have the characteristics of fast refueling and zero emissions [7,8]. Energy management strategy determines the power distribution between the fuel cell and the power battery [9], which has a significant impact on the economy and performance of the FCHEB. EMSs can be categorized into three groups: rule-based [10], optimization-based [11], and learning-based [12].

Rule-based EMSs distribute energy demand by fixed rules or fuzzy rules based on the existing engineering experience [13]. Examples include the logic threshold control strategy, thermostat strategy, and power follower strategy [14,15]. Zhang et al. [16] proposed a control strategy based on the state of charge (SOC) logic threshold. When the battery SOC is in a high state, the battery discharges, and the number of fuel cell stack operations decrease; when the battery SOC is in a low state, the number of fuel cell stack operations increases, and the battery is charged. Alexandre et al. [17] proposed a fuzzy logic controller for fuel cell hybrid electric vehicles, compared it with a dynamic programming (DP) algorithm-based control method, and improved the proposed fuzzy strategy using a genetic algorithm. These methods require little computing power, but they have the problem of poor energy-saving performance and lack adaptability to different working conditions [18].

The optimization-based strategies convert the energy allocation problem to a mathematical optimization problem and solve the optimal or suboptimal solution by means of optimization algorithms [19,20]. The optimization-based EMS can be subdivided according to whether or not it can be applied in real time. The global optimization strategy is able to obtain the optimal control sequence based on known global working conditions, such as dynamic programming and Pontryagin’s minimum principle [21,22]. Xu et al. [23] proposed a power allocation strategy based on a dynamic programming algorithm to allocate power between fuel cell engine and lithium-ion battery system to reduce operating costs. However, global optimization strategies require either a priori knowledge or high computational costs, resulting in this type of strategy being implemented only offline and, therefore, generally used only as a benchmark for comparison [24,25]. The real-time optimization strategies approach the energy management effect of the optimal solution by meeting real-time requirements. The most typical of these strategies are the equivalent consumption minimization strategy (ECMS) and model predictive control (MPC). Zeng et al. [26] proposed an adaptive ECMS that periodically updates the equivalence factor based on the predicted power through a local optimization process to converge the battery SOC and ensure fuel economy. Chen et al. [27] proposed a model predictive control-based optimization strategy for fuel cell hybrid vehicle energy management considering the variation rate of fuel cell current to improve the durability and performance of PEMFC. However, the control effectiveness of EMS based on real-time optimization is limited by low adaptability to operating conditions or high model accuracy requirements.

With the emergence of artificial intelligence algorithms, learning-based EMS is gaining more and more attention and can avoid the shortcomings of rule-based and optimization-based energy management strategies. Learning-based strategies use trial-and-error mechanisms to gradually evaluate the strategy based on interaction information and reward feedback between the strategy and the vehicle system so that the agent can eventually learn the optimized control strategy. Li et al. [28] proposed an EMS for FCHEV based on speedy Q-learning, which pre-initializes the Q-table of the RL algorithm by power distribution-related rules to improve the convergence speed. Hsu et al. [29] proposed an energy management strategy based on a reinforcement learning approach for fuel cell and battery hybrid vehicles and compared the battery SOC maintenance and fuel consumption with a fuzzy logic-based approach. Yang et al. [30] proposed a power allocation method based on Q-learning, which considers system safety, economy, and fuel cell durability to design a real-time reference path for power allocation. These studies have explored the application of reinforcement learning for energy management problems. However, deep reinforcement learning methods are introduced because, although these Q-learning- based strategies achieve some results, they fall into the dimensionality catastrophe problem when the dimensionality increases. Zheng et al. [31] proposed a deep Q-network-based power allocation method for FCHEV which considers hydrogen consumption and fuel cell degradation. Guo et al. [32] proposed an advanced dueling-double-deep Q-network-based energy management strategy to achieve a reasonable balance between system degradation and hydrogen consumption at a low economic cost. The deep Q learning (DQL) algorithm effectively solves the dimensional explosion problem. However, the discrete nature of the actions of the DQL algorithm limits the effectiveness of the control due to the continuity of energy management. Huang et al. [33] proposed an EMS based on DDPG algorithm for a range extend fuel cell hybrid vehicle to achieve optimal power allocation between fuel cell and power battery in pure electric mode and the range extend mode. DDPG introduces an actor-critical framework based on deep Q-learning compared with other deep reinforcement learning methods [34]. Zheng et al. [35] proposed a DDPG-based energy management strategy, improved the efficiency of the algorithm by prioritizing experience replay techniques, and verified the effectiveness of the proposed strategy in comparison with DP. DDPG maintains continuous states and actions and is therefore better adapted to the continuous control problem of energy management [36,37]. DDPG is now widely used in energy management problems and has achieved good performance, but DDPG overestimates Q -value, which makes it difficult to achieve optimal control [38].

Given these inherent problems, this paper proposes an EMS for FCHEBs on the basis of TD3 algorithm to reduce hydrogen costs and battery aging while considering the SOC maintenance of the lithium-ion battery and fuel cell durability. This paper contributes to related research in the following two aspects:

(1) The TD3 algorithm is used to solve the Q-value overestimation problem in the DDPG algorithm, and the proposed strategy uses real collected bus driving conditions as the training set, thus improving training efficiency and optimization.

(2) Battery aging and fuel cell power variation limits are embedded in the strategy framework to reduce hydrogen consumption and extend energy source life, thereby achieving coordination of overall vehicle operating costs and energy source durability.

The remainder of this paper is organized as follows: Section 2 presents the power system model of FCHEB. In Section 3, the TD3-based FCHEB energy management strategy is established. Section 4 compares and analyzes the proposed strategy with other benchmark strategies. Section 5 is the conclusion of this paper.

2. Configuration and Modeling

This section shows the configuration of FCHEB and describes the powertrain model, PEMFC model, battery model, and motor model of the FCHEB.

2.1. FCHEB Configuration

Figure 1 illustrates the FCHEB structure of this study. The fuel cell system transmits electricity to the DC/DC converter, and the electricity is transmitted to the DC bus after being boosted while the battery is connected directly to the DC bus. The PEMFC system consumes hydrogen to generate energy, while the battery receives electrical power from the fuel cell or brake energy recovery and discharges it when the system needs it. The PEMFC system and the power battery work together to transfer energy to the drive motor to meet the vehicle’s operating requirements when the vehicle is in motion. The main parameters of the FCHEB are listed in

Table 1. Main parameters of the FCHEB.

Components	Parameters	Values
Vehicle	Weight	12,000 kg
	Front area	8.16 m2
	Rolling radius	0.466 m
	Rolling resistance coefficient	0.0085
PEMFC System	Rated power	60 kW
Battery	Nominal energy	47.3 kWh
	Battery capacity	90 Ah
Motor	Maximum torque	1800 Nm
	Maximum speed	5000 rpm
Transmission System	Main reduction ratio	6.2
DC/DC converter	Efficiency	0.9
DC/AC inverter	Efficiency	0.95

.

2.2. Powertrain Model

The vehicle is subject to rolling resistance, air resistance, acceleration resistance, and gradient resistance during the driving process. The total demand power of the vehicle can be obtained by overcoming these resistances, which can be calculated by Equations (1) and (2) [39]. The vehicle interior is powered by the power battery and fuel cell working together, so the vehicle power balance equation is formulated as Equation (3).

$$P_{tol}=\frac{1}{{\eta }_t}(mgf\cos \alpha +\frac{1}{2}{C}_{D}A\rho v^2+mg\sin \alpha +\delta ma)\cdot v$$

(1)

$${\eta }_t={\eta }_{DC/AC}\cdot {\eta }_{EM}\cdot {\eta }_{tra}$$

(2)

$$P_{tol}=P_{FC}\cdot {\eta }_{DC/DC}+P_{bat}$$

(3)

where the $P_{tol}$ represents the total power; ${\eta }_t$ represents the efficiency of FCHEB; $m$ represents the weight of FCHEB; $g$ represents the gravitational constant; $f$ represents the rolling resistance coefficient; $C_D$ represents the aerodynamic coefficient; $A$ represents the front area; $\rho$ represents air density; $\delta$ represents the correlation coefficient of rotating mass; $a$ is the acceleration; $v$ represents the vehicle velocity; $\alpha$ represents the road grade; ${\eta }_{DC/AC}$, ${\eta }_{EM}$, ${\eta }_{tra}$, ${\eta }_{DC/DC}$ is the efficiency of DC-AC converter, electric machine, driveline, and DC-DC converter. $P_{FC}$ and $P_{bat}$ are the power of PEMFC system and power battery, respectively.

2.3. PEMFC Model

This study uses a 60 kW fuel cell system as the power unit. The fuel cell hydrogen consumption and fuel cell efficiency at the corresponding fuel cell power were measured experimentally, and the fuel cell efficiency-power data were fitted by quintic curve fitting. The characteristic curve of the fuel cell is shown in Figure 2. Hydrogen mass consumption $m_{H_2}$ of the system can be calculated as the following:

$$m_{H_2}={\displaystyle {\int }_0^t\frac{P_{FC}}{{\eta }_{FC}\cdot {\rho }_{H_2}}dt}$$

(4)

$${\eta }_{FC}=\frac{P_{FC}}{P_{H_2}}$$

(5)

where ${\rho }_{H_2}$ represents the chemical energy density of $H_2$; ${\eta }_{FC}$ represents the efficiency of the PEMFC system; and $P_{H_2}$ is the lower heating value of $H_2$.

2.4. Battery Model

This research adopts an equivalent circuit consisting of a voltage source and a resistor connected in series with the voltage source to simulate a lithium-ion battery pack, which is formulated as Equations (6)–(8) [40]. The battery charging/discharging resistance and open circuit voltage data were measured experimentally for the corresponding battery SOC, and the characteristic curves of the battery are shown in Figure 3.

$$P_{bat}=V_{ocv}{I}_{bat}-I_{bat}^2{R}_0$$

(6)

$$I_{bat}=\frac{V_{ocv}-\sqrt{V_{ocv}^2-4R_0{P}_{bat}}}{2R_0}$$

(7)

$$SoC(t+1)=SoC(t)-\frac{I_{bat}(t)\Delta t}{Q_{bat}}$$

(8)

where $V_{ocv}$ is the open-circuit voltage; $I_{bat}$ is the current; $R_0$ is the internal resistance; $SoC$ is the state of charge; $Q_{bat}$ is the nominal battery capacity.

In this study, a semi-empirical model is used to describe the capacity loss of the battery. The relationship between the capacity loss of the battery and Ah-throughput $A(c)$ at constant operating conditions can be formulated as follows:

$$Q_{loss}=B(c)\cdot \exp (\frac{-E_a(c)}{R\cdot T})\cdot A{(c)}^z$$

(9)

where $Q_{loss}$ is the capacity loss; it is generally believed that the automotive battery pack reaches the end of life at 20% capacity loss, so it takes 20; $c$ is the C-rate; $R$ is the ideal gas constant; $T$ is the absolute temperature inside the battery, which is taken as 303 $K$; $z$ represents the power law factor which is taken as 0.55; $B(c)$ is the pre-exponential factor; and $E_a$ is the activation energy, which can be obtained by [41].

The Ah-throughput $A(c)$ can be derived from Equation (9), and the equivalent number of cycles $N(c)$ can be formulated as the following:

$$A(c)={\left[\frac{Q_{loss}}{B(c)\cdot \exp (\frac{-E_a(c)}{R\cdot T})}\right]}^{\frac{1}{z}}$$

(10)

$$N(c)=\frac{3600A(c)}{Q_{bat}}$$

(11)

Hence, the governing equation of the state-of-health (SOH) can be expressed as:

$$\frac{dSOH(t)}{dt}=-\frac{\left|I_{bat}\right|}{2N(c)Q_{bat}}$$

(12)

2.5. Motor Model

The motor converts electrical energy into mechanical energy when the vehicle needs power and converts mechanical energy into electrical energy for energy recovery when the vehicle is braking. The efficiency map of the motor is illustrated in Figure 4. The efficiency of the motor can be formulated as follows:

$${\eta }_{motor}=f({\omega }_{motor},T_{motor})$$

(13)

where ${\eta }_{motor}$ is the efficiency of the motor; ${\omega }_{motor}$ is the rotational speed of the motor; $T_{motor}$ is the torque of the motor. After obtaining the speed and torque of the motor, we can determine the motor efficiency by means of a two-dimensional look-up table.

3. TD3-Based Energy Management Strategy

In this section, the TD3 algorithm and its advantages are described in detail, and the TD3-based FCHEB energy management strategy is established.

3.1. TD3 Algorithm

The TD3 algorithm is an efficient algorithm that improves on the deep deterministic policy gradient algorithm. It combines the actor-critic framework with the addition of two types of deep neural networks: the critic network $Q$ with parameters ${\theta }^Q$, and the actor network $\mu$ with parameters ${\theta }^{\mu }$. The actor network is responsible for outputting actions $a$ based on the input state $s$. The critic network obtains the current state $s$ and the actions $a$ output by the actor network and evaluates the actions accordingly to help the actor network to update. To improve the stability, the algorithm builds the target networks $Q^{\prime }$ with parameters ${\theta }^{Q^{\prime }}$ and the target networks ${\mu }^{\prime }$ with parameters ${\theta }^{{\mu }^{\prime }}$ for the actor network and the critic network, respectively. Compared to DDPG, TD3 has two critic networks, $Q_1(s,a|{\theta }^{Q_1})$ and $Q_2(s,a|{\theta }^{Q_2})$, each of which corresponds to a target network, ${Q^{\prime }}_1(s,a|{\theta }^{Q_1})$ and ${Q^{\prime }}_2(s,a|{\theta }^{Q_2})$, respectively. In addition, the target network parameters and actor network parameters are updated relatively slowly and only when the critic network updates a certain number of steps, which can improve the stability of strategy learning.

In reinforcement learning, the Temporal Difference (td) update method is usually used to accelerate the learning process of $Q$-value estimation, where the td error can be calculated as Equations (14) and (15). In the calculation, the smaller action value of these two is chosen to calculate the $Q$-value:

$$y_{\mathrm{target}}(t)=r(s_t,a_t)+\gamma \underset{i=1,2}{\min }{Q}_i^{\prime }(s_{t+1},{\widehat{a}}_{t+1}|{\theta }^{Q_i^{\prime }})$$

(14)

$$\delta (t)=y_{\mathrm{target}}(t)-Q_i(s_t,a_t|{\theta }^{Q_i})$$

(15)

where $y_{\mathrm{target}}(t)$ represents the target $Q$ network under the corresponding state $s_t$ and action $a_t$; $r$ represents the instant reward; $\gamma$ represents the discount factor to control future rewards; ${\theta }^{Q_i}$ and ${\theta }^{Q_i^{\prime }}$ are the parameters of $Q_i$ and target $Q_i^{\prime }$ network, respectively.

The control actions $a$ are obtained from the target actor network ${\mu }^{\prime }$ while adding clipped normal distribution noise based on the idea of smoothing. This process can be formulated as follows:

$${\widehat{a}}_{t+1}={\mu }^{\prime }(s_{t+1}|{\theta }^{{\mu }^{\prime }})+\xi ,\xi \sim \mathrm{clip}(\mathcal{N}(0,{\sigma }^2),-c,c)$$

(16)

The loss function is minimized by training the critic network to approximate the $Q$-value to the target $Q$-value. The loss function $L({\theta }^Q)$ and its gradient can be expressed as follows:

$$L({\theta }^{Q_i})=E[{(y_{\mathrm{target}}(t)-Q_i(s_t,a_t|{\theta }^{Q_i}))}^2]$$

(17)

$${\nabla }_{{\theta }^{Q_i}}L({\theta }^{Q_i})=E\left[(y_{\mathrm{target}}-Q_i(s_t,a_t|{\theta }^{Q_i})){\nabla }_{{\theta }^{Q_i}}{Q}_i(s_t,a_t|{\theta }^{Q_i})\right]$$

(18)

where $E(·)$ represents the mathematical expectation.

The $Q$-value can be maximized by training the actor network. The objective function $J({\theta }^{\mu })$ and its gradient can be expressed as the following:

$$J({\theta }^{\mu })=E\left[Q_1(s_t,\mu (s_t))\right]$$

(19)

$${\nabla }_{{\theta }^{\mu }}J({\theta }^{\mu })=E\left[{\nabla }_a{Q}_1(s_t,a_t|{\theta }^{Q_1}){\nabla }_{{\theta }^{\mu }}\mu (s_t|{\theta }^{\mu })\right]$$

(20)

The gradient descent update formula for the actor network and the critic network can be expressed as follows:

$${\theta }^{Q_i}\leftarrow {\theta }^{Q_i}+\alpha \cdot {\nabla }_{{\theta }^{Q_i}}L({\theta }^{Q_i})$$

(21)

$${\theta }^{\mu }\leftarrow {\theta }^{\mu }+\beta \cdot {\nabla }_{{\theta }^{\mu }}J({\theta }^{\mu })$$

(22)

where $\alpha$ is the learning rate of the critic-network; and $\beta$ is the learning rate of the actor-network.

The target networks ${Q^{\prime }}_i$ and ${\mu }^{\prime }$ are updated through soft updating by the equation,

$${\theta }^{Q_i^{\prime }}\leftarrow \tau {\theta }^{Q_i}+(1-\tau ){\theta }^{Q_i^{\prime }}$$

(23)

$${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$$

(24)

where $\tau (\tau \ll 1)$ is the factor to control updating speed.

3.2. TD3-Based EMS

Considering the operation of the FCHEB and the control objectives of the system, the state space and action space are set as follows:

$$s=[v,acc,SOC,SOH]$$

(25)

$$a=[P_{FC}|P_{FC}\in [0,60\mathrm{kW}]]$$

(26)

The optimization objective of this study is to reduce the hydrogen consumption of FCHEB and the aging of the battery based on the consideration of SOC maintenance and fuel cell durability. Based on the above requirements, the reward function is set as follows:

$$r=-\{C_{H_2}+C_{bat}+L_{SOC}+L_{FC}\}$$

(27)

$$C_{H_2}={\rho }_1\cdot [m_{{\mathrm{H}}_2}(t)]$$

(28)

$$C_{bat}={\rho }_2\cdot Q_{bat}\cdot \Delta SOH$$

(29)

$$L_{SOC}=\gamma \cdot {[SOC(t)-SO{C}_{tar}]}^2$$

(30)

$$L_{FC}=\delta \cdot \left|\Delta P_{FC}/\Delta P_{FC\max }\right|$$

(31)

where $m_{{\mathrm{H}}_2}(t)$ is the mass of hydrogen consumption; $SO{C}_{tar}$ is the target SOC value, taken as 0.6; $\Delta P_{FC}$ is the power variation of PEMFC; $\Delta P_{FC\max }$ is the maximum power variation of PEMFC, which is taken as 3000 W; and ${\rho }_1$ and ${\rho }_2$ are the unit price of hydrogen and the battery replacement cost, respectively. The unit price of hydrogen is $4/kg and the battery replacement price is $178.41/kg in this paper [9]. The corresponding weights are $\gamma$ and $\delta$, taken as 1200 and 1, respectively [42].

The architecture of TD3-based strategy is shown as Figure 5, and its pseudocode is enclosed within

Table 2. Training results of each strategy.

TD3 Algorithm
1:	Initialize critic networks $Q_i$ and actor network $\mu$ with parameters ${\theta }^{Q_i}$$\mathrm{and}{\theta }^{\mu }$
2:	Initialize target networks ${\theta }^{Q_1^{\prime }}\leftarrow {\theta }^{Q_1}$$,{\theta }^{Q_2^{\prime }}\leftarrow {\theta }^{Q_2}$$,{\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$, Initialize replay buffer $\beta$
3:	Initialize learning rate $\tau$, a random process $N$ for action exploration
4:	for episode = 1: $M$ do
5:	Initialize a random noise $\mathcal{N}(0,{\lambda }^2)\sim \frac{1}{2\lambda }\exp (-\frac{\left|x\right|}{\lambda })$ for action exploration
6:	Get initial states: $v,acc,SOC,SOH$
7:	for $t=1:T$ (time step of driving cycle) do
8:	Select action with exploration noise $a=\mu (s_t|{\theta }^{\mu })+\mathcal{N}$
9:	Execute action $a$ and observe reward $r$ and new state $s_{t+1}$
10:	Store transition tuple $(s_t,a,r,s_{t+1})$ in $\beta$
11:	Randomly sample mini-batch of $N$ transitions $(s_t,a,r,s_{t+1})$ from $\beta$
12:	$\widehat{a}\leftarrow {\mu }^{\prime }(s_{t+1}|{\theta }^{{\mu }^{\prime }})+\xi ,\xi \sim \mathrm{clip}(\mathcal{N}(0,{\sigma }^2),-c,c)$
13:	$y\leftarrow r+\gamma \underset{i=1,2}{\min }{Q}_i^{\prime }(s_{t+1},\widehat{a}|{\theta }^{Q_i^{\prime }})$
14:	Update critics ${\theta }^{Q_i}\leftarrow {\min }_{{\theta }^{Q_i}}\frac{1}{M}{\displaystyle \sum {(y_{\mathrm{target}}(t)-Q_i(s_t,a_t|{\theta }^{Q_i}))}^2}$
15:	if $t$ mod $d$ then
16:	Update $\mu$ by the deterministic policy gradient: ${\nabla }_{{\theta }^{\mu }}J({\theta }^{\mu })=\frac{1}{M}{\displaystyle \sum \left[{\nabla }_a{Q}_1(s_t,a_t|{\theta }^{Q_1}){\nabla }_{{\theta }^{\mu }}\mu (s_t|{\theta }^{\mu })\right]}$
17:	Update target networks: $\begin{array}{l}{\theta }^{Q_i^{\prime }}\leftarrow \tau {\theta }^{Q_i}+(1-\tau ){\theta }^{Q_i^{\prime }}\\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}\end{array}$
18:	end if
19:	end for
20:	End

. The TD3 system obtains the current state of the FCHEB system and the reward since the last action to output the corresponding action. The vehicle controller unit gets the action output by the TD3 system and controls the vehicle system. The vehicle system enters a new state based on the action and generates the corresponding reward for the action.

4. Results and Discussion

This section presents a comprehensive comparative analysis of the TD3-based energy management strategy, then discusses and evaluates the simulation result and performance.

4.1. Training Validation Condition

The TD3-based EMS is validated using real-world collected driving conditions as training data. The driving condition data were collected on a typical city bus route in Zhengzhou, China, and can realistically represent the driving information of the vehicle during urban road driving, which is important for the development and evaluation of energy management strategies of FCHEB. The data for real driving conditions vehicle velocity are shown in Figure 6. The mean value of the training velocity driving condition was 5.8117 m/s with a standard deviation of 5.2190. The mean value of the test velocity of driving conditions was 6.1730 m/s with a standard deviation of 5.2238.

To fully validate the performance of the proposed strategy, the TD3-based EMS is compared with the DDPG-based EMS and the DDPG-based EMS without considering the battery SOH (DDPG-NOSOH-EMS). DDPG-NOSOH-EMS does not consider the battery life constraint in the setting of the reward function. In order to verify the generality of the proposed strategy, the proposed strategy is compared with the DDPG-based EMS and the DDPG-NOSOH EMS under the standard Urban Dynamometer Driving Schedule (UDDS). Figure 7 shows the UDDS conditions. Meanwhile, in order to guarantee the reliability of the training and the credibility of the results, we take the results of four training sessions and take the mean value as the final simulation result. All simulations were performed on a computer with a processor of i5-7300HQ CPU @ 2.50GHz.

In order to reasonably compare the hydrogen consumption of each EMS, the power difference between the terminal SOC and the target SOC of each EMS is involved in the calculation of hydrogen consumption based on the hydrogen calorific conversion, considering that the fuel cell provides all the energy:

$$m_{equ}=\frac{\Delta SOC\cdot Q_{bat}\cdot 3600}{{\eta }_{FC}\cdot H_{heat}}$$

(32)

where $m_{equ}$ is the equivalent hydrogen consumption mass which reflects the SOC variation; $\Delta SOC$ represents the difference between the final SOC and the target SOC; $Q_{bat}$ is the capacity of the power battery; $H_{heat}$ is the is the heating value of hydrogen, which is taken as 120,000,000 J/kg.

4.2. Training Situation Discussion

The mean reward during training reflects the convergence efficiency and the learning ability of the algorithm. Specifically, the convergence can be judged by the distance between the mean reward of two adjacent episodes. Meanwhile, according to the setting of the reward function in this study, the closer the mean reward is to zero, the better the strategy is optimized. Figure 8 compares the mean reward of the proposed strategy and the baseline strategies, and

Table 3. Training Results of Each Strategy.

Strategy	Episodes before Convergence	Time Consumption (min)	Mature Mean Reward
TD3-based EMS	30	6.67	−0.6741
DDPG-based EMS	51	14.72	−1.0670
DDPG-NOSOH EMS	53	15.38	−0.6476

compares the training effects of these three strategies. The results show that the convergence efficiency of TD3-based EMS is significantly improved, and the training time is reduced by 54.69% and 56.63% compared with DDPG-based EMS and DDPG-NOSOH EMS, respectively. This result is due to the addition of a delayed update mechanism for the target-networks and actor-network, as well as the addition of noise in the target actions in the TD3 algorithm, which improves the stability of the algorithm and increases the convergence efficiency. Since battery aging is not considered in the reward function setting of the DDPG-NOSOH EMS, its mature mean reward is minimal. Meanwhile, because the TD3 algorithm uses Clipped Double Q-learning, it can effectively avoid the problem of overestimation of Q values by the DDPG algorithm, The learning ability of TD3-based EMS is then 36.82% higher than that of DDPG-based EMS.

4.3. Training Situation Discussion

Figure 9a shows the SOC trajectory of each strategy. The SOC of each strategy fluctuates around the target value, indicating that the strategies are able to strictly enforce the constraint of SOC maintenance in the reward function. Figure 9b–d illustrate the PEMFC power curves for each strategy. The power curves of the TD3-based EMS and DDPG-NOSOH EMS showed less fluctuation, while the SOC curves showed more fluctuation, indicating their tendency to adjust the battery output to meet the vehicle power variation. The power curve of the DDPG-based EMS is more volatile, while the SOC curve is less volatile, indicating that it prefers to adjust the PEMFC output power to satisfy the variation of vehicle power. Additionally, the power variation rate of the fuel cell for each strategy is less than 3 kW, which satisfies the limit on power variation in the reward function.

Figure 10 shows the SOH curves for each strategy. The TD3-based EMS achieves the least capacity degradation, while the DDPG-NOSOH-based EMS has the highest capacity degradation. These points illustrate the importance of adding a battery life constraint to the reward function.

4.4. Verification of Total Cost Optimization

The operating costs for each strategy under a Zhengzhou driving working condition cycle are shown in shown in

Table 4. Costs under Zhengzhou working conditions.

Strategy	Hydrogen Consumption Cost ($)	Battery Deterioration Cost ($)	Total Cost ($)
TD3-based EMS	3.51	10.81	14.32
DDPG-based EMS	3.56	11.12	14.68
DDPG-NOSOH EMS	3.69	11.32	15.01

. The total driving costs include hydrogen consumption cost and battery degradation cost. The hydrogen consumption cost takes into account the equivalent hydrogen consumption caused by the different terminal SOC of each strategy. Considering the table data, it can be concluded that for each strategy, the cost of battery aging is over 75% of the total cost. The results show that, compared with the DDPG-based EMS and the DDPG-NOSOH-based EMS, the TD3-based EMS indicated a 1.40% and 4.88% decrease in hydrogen consumption cost, a 2.91% and 4.51% decrease in battery deterioration cost, and a 2.45% and 4.60% decrease in total cost, respectively.

Table 5. Costs under UDDS working conditions.

Strategy	Hydrogen Consumption Cost ($)	Battery Deterioration Cost ($)	Total Cost ($)
TD3-based EMS	3.54	9.16	12.70
DDPG-based EMS	3.56	9.70	13.26
DDPG-NOSOH EMS	3.62	9.95	13.57

shows the operating costs for each strategy under a UDDS cycle, where the hydrogen consumption costs are corrected. The comparison results show that the proposed strategy is optimized in terms of hydrogen consumption cost and battery deterioration cost.

5. Conclusions

In this paper, a fuel cell hybrid electric bus energy management strategy based on TD3 algorithm is proposed to optimize the driving cost of fuel cell hybrid electric buses. Using the real collection data of Zhengzhou buses as training data, the effectiveness of the TD3-based EMS is verified by comparing it with other strategies in terms of hydrogen consumption costs, battery degradation costs, and total cost. The major conclusions of this paper are as follows:

(1) The cost of battery degradation accounts for a large proportion of the vehicle operating cost, and it is undesirable to neglect the battery degradation.

(2) By adding new mechanisms to the DDPG algorithm, the training effect of TD3-based EMS is enhanced. Compared with the DDPG-based EMS using the same reward function, the training efficiency and learning ability of the TD3-based EMS are 54.69% and 36.82% higher, respectively.

(3) Under Zhengzhou bus driving conditions, the overall cost of TD3-based EMS was decreased by 2.45% and 4.60% compared to DDPG-based EMS and DDPG-NOSOH EMS, respectively.

In the future, we will consider integrating a cyber physical system (CPS) into the EMS framework when designing an energy management strategy. Information on vehicle operation is obtained through cyber physical systems to optimize the controller’s control decisions. In addition, in terms of overall vehicle operating costs, an analysis of fuel cell aging costs will be added. Meanwhile, we will actively focus on the latest deep reinforcement learning algorithms for their application in the field of energy management.