Deep Reinforcement Learning-Based Real-Time Joint Optimal Power Split for Battery–Ultracapacitor–Fuel Cell Hybrid Electric Vehicles

Abstract

Hybrid energy storage systems for hybrid electric vehicles (HEVs) consisting of multiple complementary energy sources are becoming increasingly popular as they reduce the risk of running out of electricity and increase the overall lifetime of the battery. However, designing an efficient power split optimization algorithm for HEVs is a challenging task due to their complex structure. Thus, in this paper, we propose a model that jointly learns the optimal power split for a battery/ultracapacitor/fuel cell HEV. Concerning the mechanical system of the HEV, two propulsion machines with complementary operation characteristics are employed to achieve higher efficiency. Additionally, to train and evaluate the model, standard driving cycles and real driving cycles are employed as input to the mechanical system. Then, given the inputs, a temporal attention long short-term memory model predicts the next time step velocity, and through that velocity, the predicted load power and its corresponding optimal power split is computed by a soft actor–critic deep reinforcement learning model whose training phase is aided by shaped reward functions. In contrast to global optimization techniques, the local velocity and load power prediction without future knowledge of the driving cycle is a step toward real-time optimal energy management. The experimental results show that the proposed method is robust to different initial states of charge values, better allocates the power to the energy sources and thus better manages the state of charge of the battery and the ultracapacitor. Additionally, the use of two motors significantly increases the efficiency of the system, and the prediction step is shown to be a reliable way to plan the HESS power split in advance.

1. Introduction

The basic operating principle of internal combustion engine (ICE) vehicles involves transforming energy from fossil fuels into thermal energy. In this combustion, the gases generated are released into the atmosphere with a negative impact on the environment and human health. In particular, the transportation sector contributed 29% of greenhouse gas emissions (GHG) to total United States GHG emissions in 2019 [1]. Furthermore, there is a growing concern regarding the scarcity of fossil fuels and the need to implement a sustainable economy based on renewable energy sources. The Sustainable Development Goals (SDGs) established in 2015 by the United Nations General Assembly have sustainable energy at its core. The seventh goal specifically emphasizes the need for renewable energy sources [2]. In light of the growing concern of society with environmental issues, the development of battery electric vehicles (BEVs) is a step toward the fulfillment of the seventh SDG.

The engine is the primary distinction between a traditional internal combustion engine (ICE) car and an electric vehicle (EV). The latter is powered by an electric motor that works by transforming chemical energy stored in rechargeable batteries into electrical energy. The generated electricity is then transformed into mechanical and kinetic energy. These new models of vehicles also have multiple advantages when compared to ICE ones. EVs are efficient, do not emit tailpipe pollutants, are quiet and produce little noise pollution. In addition, recharging an EV is cheaper than refueling an ICE car [3,4].

Electric vehicles include battery electric vehicles (BEVs), hybrid electric vehicles(HEVs), plug-in hybrid electric vehicles(PHEVs), and fuel cell hybrid electric vehicles (FCHEVs), depending on the energy source for the vehicle [5,6].

Fuel cells are considered an ideal energy source for electric vehicles because of their high efficiency and zero emission [7]. However, there is very little charging infrastructure for fuel cells compared to charging stations for EVs and gas stations for ICEs. Moreover, the cost of charging a fuel cell is much higher than that of charging an electric vehicle. Because of this shortcoming of fuel cells, FCHEVs require additional energy sources.

Batteries are the most widely used energy source in EVs because they have high energy density [8,9]. However, they have disadvantages such as long charging times, high prices, and high temperature sensitivity. Therefore, recently, solid-state type batteries with better performance and shorter charging time than conventional batteries have been continuously studied [10,11].

Ultracapacitors (UC) have high power density and very long lifetimes and are not affected by temperature; so they are suitable for devices with high peak currents [12]. However, low energy density, the need for voltage balancing and high self-discharging are major drawbacks. One of the most promising solutions proposed for FCHEVs is a hybrid energy storage system (HESS) consisting of three energy sources: a fuel cell (FC), a battery, and an ultracapacitor [13].

Since the HESS structure for EVs is diverse, a different energy management system (EMS) control strategy is required according to the HESS structure [14]. One of the most common HESS structures consists of a battery and a UC [15]. Among other existing studies, there are studies that reviewed a HESS consisting of an FC and a battery [16] and an FC and a UC [17]. As a more complex structure, there have been studies on hybrid energy storage systems that include all three of them: a battery, a UC, and an FC [18].

In addition, there are some studies [19,20] of EMS considering a complex HESS structure in which not only multiple energy storage units but also more than one propulsion machine are connected. To improve power performance and propulsion efficiency, two propulsion machines with complementary torque–speed characteristics can be used in the EV powertrain.

The energy management systems of electric vehicles with hybrid energy storage systems are largely classified into two types: rule-based and optimization-based [21]. Rule-based EMSs are divided into deterministic rule-based EMSs and fuzzy rule-based EMSs according to the characteristics of the rules used. In addition, optimization-based EMSs can be classified into global optimization and real-time optimization EMSs according to the level of information of the driving condition. Global optimization-based methods have the advantage of finding the global optimum of the problem, but they demand previous knowledge of the entire driving cycle, which would be unfeasible in a real-time optimization scenario.

Many previous studies focused on convex optimization techniques [19,20,22,23]. However, convex optimization of complex systems is very difficult due to the excessive number of parameters to consider and the difficulty of linearizing all systems. Therefore, a neural network (NN)-based machine learning algorithm was proposed by [20] to solve the multi-purpose energy problem for a dual-motor battery/FC/UC. The control was performed through two steps. First, the load power was divided between the two complementary motors, and then the load power was split between the battery pack, the FC and the UC in the HESS. For the HESS power split, a convex optimization technique was used to generate optimal target values which became the training data for the NN. When employing convex optimization, the results showed that the battery life was extended by about 5 years in contrast to the NN model which extended the battery life by about 92.5% of 5 years.

Another technique that has been emerging in the area of HEV EMSs is deep reinforcement learning (DRL) [24,25,26,27,28]. They explore different DRL techniques such as deep Q-learning (DQN), soft actor–critic (SAC) and deep deterministic policy gradient (DDPG) for different HEV topologies. Particularly, by applying reinforcement learning using real driving data, energy savings of about 16% compared to the existing binary control strategy were confirmed [28]. The energy efficiency was improved by looking at the existing driving cycle of one’s vehicle and dividing the power output level of the ICE into 24 levels through reinforcement learning, instead of just following the rules set by the EMS.

However, we found only one work that addresses the dual-motor battery/UC/FC HEV EMS control strategy, due to their complex structure [20], which makes use of convex optimization and machine learning techniques, not reinforcement learning. Additionally, even though the authors in [27] develop a reinforcement learning-based model for an FC/battery/UC HEV, they use a deep-Q network model and do not explore the use of multiple motors nor the forecast of the future load power. Thus, in this paper, a real-time deep reinforcement learning (SAC)-based EMS control strategy for a HESS of a vehicle that has three energy sources, battery, FC and UC, and two complementary motors is proposed. First, a method for real-time velocity and load power prediction using SDCs and RDCs is proposed. This prediction step allows the EMS to better plan the use of resources of the HEV and prevent it from running out of electricity. Given the predicted load power, the SAC model is then used to efficiently distribute energy in the HESS. Through the experimental results, it was confirmed that compared to traditional rule-based methods the proposed method can better allocate the energy sources power and thus achieve good FC efficiencies and better manage the SOCs of the battery and the ultracapacitor. The model is also shown to be robust, as it can handle different initial values of SOC while satisfying all the constraints of the system. Moreover, the addition of reward shaping to the training phase of the SAC agent accelerated its convergence. Additionally, it can be seen that the use of two complementary motors leads to a big improvement in the efficiency of the vehicle compared to a single-motor architecture. Finally, the results of the prediction step show that the method is reliable to forecast the future speeds and load power and, therefore, aid the model in allocating the resources of the HEV.

2. Preliminaries

2.1. Overall Procedure for a Reinforcement Learning-Based Energy Management Strategy (EMS)

Figure 1 illustrates the proposed method. The inputs of the neural network are the five past velocity and acceleration values of either standard driving cycles (SDCs) or real driving cycles (RDC). Given those values, the neural network predicts the future velocity and acceleration for one time step ahead, which will be used to compute the load powers of the two electrical motors. The load powers of these motors will be sent into the EMS, which will distribute the load power between the battery, the fuel cell, and the ultracapacitor using deep reinforcement learning (DRL).

2.2. Input Data

For the input, both SDCs and RDCs were used. Particularly, the worldwide harmonized light vehicles test cycles (WLTC) and the urban dynamometer driving schedule (UDDS) were used. Despite their ease of use, they do not represent the reality well enough; so they were mainly used for model validation and performance evaluation purposes. On the other hand, RDCs overcome these limitations because of the higher degree of complexity associated to them. There are personal elements, such as the driver’s behavior, which is hard to model because of its time-varying nature. There are also factors that do not depend on the driver, such as the weather conditions (snowstorm, heavy rain, flooding), traffic signals and traffic conditions (construction and accidents may slow down the traffic). The RDCs data, collected by an On-Board Diagnostics (OBD)-II dongle connected to one of the author’s vehicles, are the same as the data used in Hong et al. [29]. The four RDCs employed in this paper are shown in Figure 2.

2.3. Neural Network for Velocity Forecast

The problem that the neural network is trying to solve can be modeled as a univariate time series; that is, there is only one time-varying variable, which is the velocity of the vehicle. Despite the existing challenges related to time series forecast, it is an important field of study because knowing the future allow us to better plan short-term and long-term goals. Particularly, in our work, we predict the velocity to optimally allocate the available resources so that the energy storage components may achieve maximum efficiency, their lifetime may be maximized and the vehicle may not run out of electricity during the trip.

Based on the previous work of Hong et al. [29], which used the same RDC dataset as this present paper and evaluated different time series and deep learning models for velocity prediction (seasonal auto regressive integrated moving average (SARIMA), recurrent neural network (RNN), gated recurrent unit (GRU) and long short-term memory (LSTM)), we decided to use the attention-based LSTM model with step size $T=5$ and hidden state size $m=32$ proposed by the authors, as it outperformed the other models that were tested.

2.4. Vehicle System Overview

The vehicle is mainly composed of the HESS and the mechanical components shown in Figure 3. The HESS is composed of three energy sources (battery/ultracapacitor/fuel cell), and the HEV has two complementary propulsion machines.

2.5. Power Split Optimization for the Propulsion Machines

The mechanical model of the vehicle may be computed by Equations (1)–(14), and its parameters are given in

Table 1. Mechanical model parameters.

Parameters	Description	Values
m	Mass of the vehicle (kg)	1650
$r_{wh}$	Wheel radius (m)	0.33602
g	Gravity acceleration (m/s${}^2$)	9.81
$\rho$	Air density	1.2
A	Front area of vehicle (m${}^2$)	2.12344
$\alpha$	Angle of driving surface (rad)	0
$a_0$	Rolling resistance coefficient	$8.8·{10}^{-3}$
$a_1$	Rolling resistance coefficient	$-6.42·{10}^{-5}$
$a_2$	Rolling resistance coefficient	$9.27·{10}^{-6}$
$a_3$	Rolling resistance coefficient	$-3.3·{10}^{-7}$
$a_4$	Rolling resistance coefficient	$6.68·{10}^{-11}$
$a_5$	Rolling resistance coefficient	$-4.46·{10}^{-11}$
$C_d$	Aerodynamic drag coefficient	0.24
$G_r$	Gearbox ratio	5.5

. The acceleration values needed for the equations come from Section 2.3 by differentiating the forecast velocity values. Such acceleration values are then used to compute the total force $F_t$ acting on the vehicle, given by (1), consisting of the rolling resistance $F_{rr}$, the aerodynamic drag $F_d$, the grading resistance $F_{gr}$ and the $F_{la}$. Given the total force, the load power and the power loss for each motor may be computed according to the motor efficiencies. Then, combining all the Equations (1)–(14) with the power split optimization procedure proposed by [19], the optimal power split and losses for the two motors may be computed.

In this paper, the motors chosen are the Toyota Camry MG1 [30] and the UQM PowerPhase 125 [31]. Their parameters are shown in

Table 2. Parameters of the motors.

Motor	Parameters	Values
Toyota Camry MG1	Motor peak power rating (kW)	105
	Motor peak torque rating (Nm)	270
	Top rotational speed (rpm)	14,000
	Motor mass (kg)	41.7
UQM PowerPhase 125	Motor peak power rating (kW)	125
	Motor peak torque rating (Nm)	300
	Top rotational speed (rpm)	8000
	Motor mass (kg)	41

and their efficiency curves may be seen in Figure 4. Finally, Figure 5 shows the output of the method (load power and losses) for the two motors for the four RDCs shown in Figure 2.

$$F_t=F_{rr}+F_{ad}+F_{gr}+F_{la}$$

(1)

$$F_{rr}=C_{r}mgcos\left(\alpha \right)$$

(2)

$$C_r=a_0+a_1·v+a_2·v^2+a_3·v^3+a_4·v^4++a_5·v^5$$

(3)

$$F_{ad}=\frac{1}{2}\rho A{C}_d{v}^2$$

(4)

$$F_{gr}=mgsin\left(\alpha \right)$$

(5)

$$F_{la}=\delta ma$$

(6)

$$T_t=F_t\times r_w$$

(7)

$$P_t=F_t\times v$$

(8)

$${\omega }_w=\frac{v}{r_w}$$

(9)

$$\tau =\frac{r_{\mathrm{wh}}·F_t}{G_r}$$

(10)

$${\omega }_m=\frac{v}{r_{\mathrm{wh}}}{G}_r$$

(11)

$$P_{\mathrm{Load}}=\tau {\omega }_m$$

(12)

$$P_{\mathrm{loss},i}\left(t\right)=\left\{\begin{array}{ccc}\omega \left(t\right)·{\tau }_i\left(t\right)·\left(1-{\eta }_i\left(t\right)\right)\hfill & \mathrm{if}\hfill & {\tau }_i\left(t\right)\ge 0\hfill \\ \omega \left(t\right)·{\tau }_i\left(t\right)·\frac{\left(1-{\eta }_i\left(t\right)\right)}{{\eta }_i\left(t\right)}\hfill & \mathrm{if}\hfill & {\tau }_i\left(t\right)\le 0\hfill \end{array}\right\},\mathrm{i}=\{1,2\} \mathrm{is} \mathrm{the} \mathrm{motor} \mathrm{index}$$

(13)

$$\mathrm{Efficiency}\left(\eta \right)=\mathrm{lookup}2\mathrm{d}\left(\mathrm{Torque}\right(\mathrm{Nm}),\mathrm{Speed}(\mathrm{rpm}\left)\right)$$

(14)

2.6. Electrical Model of the Hybrid Energy Storage System (HESS)

The battery and the ultracapacitor packs adopted are based on the ones proposed by [19]. They are composed of, respectively, multiple K2 High Capacity Lithium iron phosphate (LiFePO4) 22650P battery cells [32] and multiple Maxwell BCAP1500 ultracapacitors [33]. Their basic models are given, respectively, by Figure 6 and Figure 7; the parameters are given by

Table 3. Battery parameters.

Parameters	Description	Values
R	Internal resistance	9 m$\mathsf{\Omega }$
$N_s$	Number of serial cells	94
$Np$	Number of parallel branches	36
$V_T$	Battery voltage (V)	300
$R_n,n=1,2,\dots ,N_s$	Equivalent resistance of serial cells	846 m$\mathsf{\Omega }$
$R_T$	Total equivalent resistance	23.5 m$\mathsf{\Omega }$
$V_{cell}$	Cell voltage (V)	3.2
$E_{cell}$	Maximum energy stored in a cell (Wh)	8.32
$C_{cell}$	Capacity of a cell (Ah)	2.6
$E_T$	Total energy stored (kWh)	28.2
$C_T$	Total capacity (Ah)	93.6
$SO{C}_{bat}$	SOC of the battery	-
$i_{bat}$	Current of the battery	-

and

Table 4. Ultracapacitor parameters.

Parameters	Description	Values
R	Internal resistance	0.47 m$\mathsf{\Omega }$
$N_s$	Number of serial cells	222
$V_T$	Battery voltage (V)	600
$R_T$	Total equivalent resistance	103.34 m$\mathsf{\Omega }$
$C_{cell}$	Capacitance of a cell (F)	1500
$V_{cell}$	Cell voltage (V)	2.7
$E_{cell}$	Maximum energy stored in a cell (Wh)	1.52
$E_T$	Total energy stored (kWh)	0.338
$C_T$	Total capacitance (F)	6.76
$SO{C}_{UC}$	SOC of the battery	-
$i_{UC}$	Current of the battery	-

, and the equations used to compute the parameters are given by Equations (15)–(22) and (23)–(29).

Finally, the fuel cell (FC) and the DC/DC converter were modelled based on the system proposed in [20]. The DC has a maximum output of 35kW, the parameters for the FC are given in

Table 5. FC and DC/DC converter parameters.

Parameters	Description	Values
$P_{nom,fc}$	FC nominal power	8 kW
$a_1$	Slope (if ${\displaystyle \frac{P_{fc}}{P_{fc,nom}}}\le 0.15$)	2
$b_1$	Intercept (if ${\displaystyle \frac{P_{fc}}{P_{fc,nom}}}\le 0.15$)	0.4
$a_2$	Slope (if ${\displaystyle \frac{P_{fc}}{P_{fc,nom}}}\ge 0.15$)	−0.040625
$b_2$	Intercept (if ${\displaystyle \frac{P_{fc}}{P_{fc,nom}}}\ge 0.15$)	0.620625
${\eta }_{fc}$	FC efficiency	Equation (30)
$P_{fc,loss}$	Loss in the FC	Equation (31)
${\eta }_{inv}$	Inverter efficiency	0.97
$P_{inv,loss}$	Loss in the inverter	Equation (32)

and the FC stack’s efficiency and power loss may be computed according to Equations (30) and (31).

Two inverters for two propulsion machines were modelled based on the datasheet [34]. These inverters are insulated gate bipolar transistors (IGBTs) with a based pulse width modulated (PWM) inverter. We assumed that inverter efficiency is 97%.

$$V_T=N_s·V_{cell}$$

(15)

$$R_n=N_s·R;n=1,...,94$$

(16)

$$R_T={\displaystyle \frac{R_n}{N_p}}$$

(17)

$$E_{cell}=V_{cell}·C_{cell}$$

(18)

$$C_T=C_{cell}·N_p$$

(19)

$$E_T=E_{cell}·N_p·N_s$$

(20)

$$i_{bat}=\frac{V_T-\sqrt{V_T^2-4R_T}}{2R_T}$$

(21)

$$So{C}_{bat}=So{C}_{bat,prev}+\frac{\beta \int i_{bat}dt}{C_T}$$

(22)

$$V_T=N_s·V_{cell}$$

(23)

$$R_T=N_s·R$$

(24)

$$C_T={\displaystyle \frac{C_{cell}}{N_s}}$$

(25)

$$E_{cell}=0.5·C_{cell}·V_{cell}^2$$

(26)

$$E_T=E_{cell}·N_s$$

(27)

$$i_{UC}=\frac{V_T-\sqrt{V_T^2-4R_T}}{2R_T}$$

(28)

$$So{C}_{UC}=So{C}_{UC,prev}+\frac{\beta \int i_{UC}dt}{C_T}$$

(29)

$${\eta }_{fc}=a·{\displaystyle \frac{P_{fc}}{P_{fc,nom}}}+b$$

(30)

$$P_{loss,fc}=P_{fc}·(1-{\eta }_{fc})$$

(31)

$$P_{loss,inv}=P_{inv}·(1-{\eta }_{inv})$$

(32)

$$P_{PM}=P_{inv}·{\eta }_{inv}$$

(33)

3. Soft Actor–Critic (SAC) for HESS Power Split Optimization

3.1. Problem Overview

Reinforcement learning (RL) at its core involves an agent trying to learn decision making and control by interacting with an environment. Such interaction happens through actions taken by the agent, which can modify the environment and cause a state transition. As every action has an impact on the environment, each yields rewards to encourage good actions and discourage bad ones. The choice of action for each state depends on the policy of the agent, which maps the states to the probabilities of choosing each action given a specific state.

Therefore, the problem of power split optimization for the presented HESS can be modeled as a deep reinforcement learning problem. As shown in the next sections, the agent of our proposed model must learn the best policy possible to maximize the rewards of the system as a whole. Moreover, as it is not possible to map all the possible state–action pairs of the environment to their respective rewards, we use deep neural networks (DNN) to approximate them. Finally, all the code related to the deep reinforcement learning-based model was implemented using the Python programming language and the TensorFlow’s TF-Agents framework.

3.2. Agent, Environment, Action and State

In our case, we have the HESS, which is the agent inside the HEV, which is the environment. The HEV is supposed to move for a specified amount of time according to a driving cycle while varying velocity, optimizing the fuel cell’s efficiency and minimizing the battery power magnitude and variation. However, initially the HEV does not know how to distribute the load power computed in Section 2.5 between the three different energy storage systems (battery, ultracapacitor and fuel cell). The task of power split is performed by the HESS, which must decide among infinite possible combinations of the optimal battery, ultracapacitor and fuel cell powers. Each of the combinations chosen is an action performed by the agent, which leads to a state transition. Thus, in this paper, the state and the action spaces are given, respectively, by $S=\{P_{load};SO{C}_{bat};SO{C}_{uc}\}$ and $A=\{P_{bat,\%},P_{UC,\%},P_{FC,\%}\}$. The actions are not absolute power values but continuous values ranging from zero to one, which are transformed into percentages summing up to one through the softmax function. For instance, if $P_{load}=10$ kW and $\{P_{bat,\%},P_{UC,\%},P_{FC,\%}\}$ = {0.8, 0.5, 0.2}, the softmax function outputs approximately $\{0.44,0.32,0.24\}$, which would represent in absolute power values $\{P_{bat},P_{UC},P_{FC}\}$ = {4.4 kW, 3.2 kW, 2.4 kW}.

3.3. Rewards and Penalties

The learning process of the agent is driven by rewards and penalties depending on how good the agent’s actions are. In our case, the agent should ideally satisfy all the following criteria (35)–(41), which are based on [20]. The parameters related to the constraints may be found in

Table 6. Optimization constraints parameters.

Parameters	Description	Values
$P_{min,bat}$	Minimum battery power	−30 kW
$P_{max,bat}$	Maximum battery power	30 kW
$SO{C}_{min,bat}$	Minimum battery SOC	0.2
$SO{C}_{max,bat}$	Maximum battery SOC	0.9
$P_{min,bat\_diff}$	Minimum battery power difference	−9 kW
$P_{max,bat\_diff}$	Maximum battery power difference	9 kW
$P_{min,DC}$	Minimum DC/DC converter power	−35 kW
$P_{max,DC}$	Maximum DC/DC converter power	35 kW
$P_{min,FC}$	Minimum FC power	0
$P_{max,FC}$	Maximum FC power	8 kW
$P_{min,UC}$	Minimum ultracapacitor power	−30 kW
$P_{max,UC}$	Maximum ultracapacitor power	30 kW
$SO{C}_{min,UC}$	Minimum ultracapacitor SOC	0.5
$SO{C}_{max,UC}$	Maximum ultracapacitor power	1

, and the objective function to be optimized is shown in Equation (34), where the coefficients (a) to (f) are positive penalty coefficients.

$$g=a·P_{bat}+b·P_{UC}+c·P_{FC}+d·SO{C}_{bat}^2+e·SO{C}_{UC}^2+f·{\eta }_{FC}^2$$

(34)

$$P_{min,bat}\le P_{bat}\left(t\right)\le P_{max,bat}$$

(35)

$$P_{min,bat\_diff}\le P_{bat}\left(t\right)-P_{bat}(t-1)\le P_{max,bat\_diff}$$

(36)

$$P_{min,DC}\le P_{DC}\left(t\right)\le P_{max,DC}$$

(37)

$$P_{min,FC}\le P_{FC}\left(t\right)\le P_{max,FC}$$

(38)

$$P_{min,UC}\le P_{UC}\left(t\right)\le P_{max,UC}$$

(39)

$$P_{bat}\left(t\right)+P_{FC}\left(t\right)-P_{DC}\left(t\right)=P_{M_1}\left(t\right)$$

(40)

$$P_{UC}\left(t\right)+P_{DC}\left(t\right)=P_{M_2}\left(t\right)$$

(41)

Considering the constraints (35)–(41), terminal states were designed. The terminal states and actions would be a $SO{C}_{bat}$, $SO{C}_{uc}$, $P_{bat}$, $P_{uc}$ and $P_{FC}$ smaller or greater than the minimum or maximum values allowed according to Table 6. Thus, whenever the vehicle reached an illegal state or tried to perform an illegal action, the environment was reset, and the agent received a penalty proportional to the length of the driving cycle. As the length of the driving cycles used were all less than 2000, the agent receives a penalty of 1000, and the environment is reset. On the other hand, if the states and the actions chosen are valid, the vehicle receives a reward according to $r=SO{C}_{bat}^2+SO{C}_{uc}^2+{\eta }_{fc}^2+g\left(x\right)$, where $g\left(x\right)=1-{\left({\displaystyle \frac{d}{d_{max}}}\right)}^p$ is the output of the shaped reward function as explained below.

Reward shaping is a technique that involves changing the structure of a sparse reward function to offer more regular feedback to the agent [35] and thus accelerate the learning process. Figure 8 shows an example of a sparse and a shaped reward function.

In our paper, we designed three different reward functions and one penalty function. Equations (42)–(44) show the reward functions, and Equation (45) represents the penalty function for different values of d and $d_{max}$ of the equation $g\left(x\right)=1-{\left({\displaystyle \frac{d}{d_{max}}}\right)}^p$. The value of p was set to 0.4 after performing hyperparameter tuning.

All the four functions were activated depending on the SOC of the ultracapacitor to encourage higher or lower power from the three energy sources. Their thresholds and descriptions are shown in

Table 7. Shaped reward functions.

Function	Threshold	Description
$g_1\left(x\right)$	Activated when $SO{C}_{UC}>0.65$ or ($SO{C}_{UC}>0.65$ and $P_{load}<0$)	Encourages high UC power or regenerative power
$g_2\left(x\right)$	Activated when $SO{C}_{UC}<0.65$	Encourages high FC power
$g_3\left(x\right)$	Activated when $SO{C}_{UC}<0.65$	Encourages high battery power
$g_4\left(x\right)$	Activated when $SO{C}_{UC}<0.65$	Encourages low UC power

.

$$g_1\left(x\right)=1-{\left({\displaystyle \frac{P_{max,UC}-P_{UC}}{P_{max,UC}}}\right)}^{0.4}$$

(42)

$$g_2\left(x\right)=1-{\left({\displaystyle \frac{P_{max,FC}-P_{FC}}{P_{max,FC}}}\right)}^{0.4}$$

(43)

$$g_3\left(x\right)=1-{\left({\displaystyle \frac{P_{max,bat}-P_{bat}}{P_{max,bat}}}\right)}^{0.4}$$

(44)

$$g_4\left(x\right)=-\left(1-{\left({\displaystyle \frac{P_{max,UC}-P_{UC}}{P_{max,UC}}}\right)}^{0.4}\right)$$

(45)

3.4. Soft Actor–Critic (SAC)

The essence of solving a reinforcement learning problem lies in optimizing the trade-off between exploration and exploitation. In contrast to supervised learning, in RL there are no labels, and the agent must learn to satisfy all the rules of the environment simply by exploration. During the training stage, the model will solely perform random actions and gradually find an optimal balance between exploration and exploitation, thus the optimal policy. On the other hand, during the testing stage, the model will only perform exploitation. That is, it will act according to the optimal policy learned during the training stage.

The soft actor–critic (SAC) model [36] is an off-policy method that uses DRL to find the optimal balance between exploration and exploitation by maximizing both the reward and the entropy of the system. By maximizing the entropy, the model is encouraged to keep exploring and thus assign similar probabilities to actions with similar action values and not assign excessively large probabilities to a specific set of actions. On the other hand, by maximizing the rewards, the model will strive toward finding the optimal policy. Therefore, given the large action and state spaces of our model (see Figure 5 and Table 6), we believe that the SAC model would be appropriate to learn the ideal power split algorithm for the HESS.

The architecture chosen for our SAC agent is a two-network design without shared features between them: one for the actor and one for the critic. Their parameters after hyperparameter tuning are shown in

Table 8. SAC agent parameters.

Parameters	Values
Activation function	ReLU
Discount factor	0.999
Learning rate	$3\times {10}^{-4}$
Batch size	2048
Number of hidden layers	2
Number of hidden units per layer	256
Target smoothing coefficient	0.005
Target update interval	1
Gradient steps	1
Replay buffer size	1,000,000

.

4. Experimental Results

4.1. SAC Agent Training Phase

The training phase of the SAC agent was performed using the WLTC class 3 data and the parameters shown in Table 8. Figure 9 shows the rewards over 1,400,000 iterations, while Figure 10 represents the average rewards over 10,000 steps. It may be seen that the average rewards converge to the value of about 800.

4.2. SAC Agent Evaluation with SDCs

To evaluate the proposed method, a rule-based power split technique was employed in the HESS for the WLTC class 1 standard driving cycle. The power split rule changed based on the sign of the load power. For positive load powers, ratios of $\{P_{bat},P_{UC},P_{FC}\}=\{40\%P_{load},40\%P_{load},20\%P_{load}\}$ were used, and in the case of regenerative braking, ratios of $\{P_{bat},P_{UC},P_{FC}\}=\{50\%P_{load},50\%P_{load},0\%P_{load}\}$ were used. The results are shown in the Figure 11, Figure 12 and Figure 13.

Both techniques have similar results in terms of power split, but it can be seen that the rule-based technique struggles with managing the SOCs and the currents of the ultracapacitor and the battery. The initial SOCs are 0.9, but after the 1023 s of simulation, the final SOCs of the ultracapacitor are about 62.8% and 46.9% for the DRL and rule-based technique, respectively, as shown in Figure 13. The latter value is below the acceptable value because the theoretically minimum allowed SOC for the ultracapacitor is 50%, as shown in Table 6 due to the approximately linear discharge of the ultracapacitor.

Additionally, the SAC model is far more robust than the rule-based one. Figure 14 shows the SOCs of the battery and the UC when their initial values are 0.7. The SAC model is able to find the optimal power split while satisfying the constraints from Table 6, in contrast to the rule-based technique.

Finally, the DRL model also has the advantage of being able to perform well in new data without the need for retraining. It is shown in Section 4.3 and Section 4.4 that the model can obtain good results, even for different data on which it was never trained.

4.3. SAC Agent Evaluation with RDCs

As explained in Section 2.2, four RDCs were employed to evaluate the model.

Table 9. Model results without prediction step.

RDC	${\mathit{SOC}}_{\mathit{UC}}$				${\mathit{SOC}}_{\mathit{bat}}$				Mean ${\mathit{\eta }}_{\mathbf{FC}}$	Motors Efficiency
	Initial	Final	Min	Max	Initial	Final	Min	Max		M1 Only	M1 + M2	Improvement
1	55%	61.2%	50.7%	61.2%	55%	54.5%	54.5%	55%	55.8%	69.6%	79.8%	14.7%
	90%	81.5%	77.3%	90%	90%	89.7%	89.6%	90%	56%
2	55%	62.7%	50.4%	62.7%	55%	54.5%	54.4%	55%	56.2%	73.8%	80.5%	9.1%
	90%	82%	75.2%	90%	90%	89.7%	89.6%	90%	56.4%
3	55%	62.5%	50.2%	62.9%	55%	54.4%	54.3%	55%	57.3%	68.3%	80.3%	17.6%
	90%	82%	75.3%	90%	90%	89.6%	89.5%	90%	57.3%
4	55%	61.3%	50%	61.5%	55%	54.5%	54.4%	55%	55.9%	70%	81.3%	16.1%
	90%	81.5%	75.6%	90%	90%	89.7%	89.6%	90%	56.2%

shows the results obtained by the proposed method when prediction of the speed and load power is not considered. First, it is interesting to analyze the $SO{C}_{UC}$. Two initial values were considered: 55% and 90%. In general, when the initial $SO{C}_{UC}=55\%$, the model could perform the optimal power split while respecting the constraints from Table 6, as the minimum $SO{C}_{UC}$ did not go below 50%. Additionally, it may be seen that the model focused on recharging the UC through regenerative braking, as the final $SO{C}_{UC}$ was greater than the initial. On the other hand, when the initial $SO{C}_{UC}=90\%$, the model allocated more power to the UC instead of recharging it, as the final $SO{C}_{UC}$ of approximately 82% is smaller than the initial $SO{C}_{UC}$.

Second, considering that the minimum and maximum efficiencies ${\eta }_{FC}$ are, respectively, 40% and 62.1%, the obtained results in the range of 55.8% to 57.3% were satisfactory. We have also plotted the FC powers related to the RDC1 for $SO{C}_{UC}=55\%$ and $SO{C}_{UC}=90\%$ in Figure 15 to better analyze the FC results. One thing to note is that there was no significant difference between the plots. However, one would expect that the model would allocate more power to the FC for low values of $SO{C}_{UC}$ to reduce the use of the UC and prevent it from going below the minimum of $50\%$. This means that the model could be a little bit further optimized to improve the power split method.

Third, compared to the Toyota Camry MG1-only structure, there is a significant efficiency improvement when two motors are used. The high improvement of 17.6% is achieved because the two motors are complementary.

4.4. Evaluation of the Prediction Method with RDCs

In this section, we evaluate the velocity and load power prediction method using the model described in Section 2.3 and Section 2.5, along with the SAC agent. Figure 16, Figure 17 and Figure 18 show the results obtained. By analyzing the figures, it can be seen that the predicted powers and SOCs are not too far from the actual ones. Additionally,

Table 10. Model results with prediction step.

RDC	MAE
	Speed (m/s)	${\mathit{P}}_{\mathit{load}}\left(\mathit{W}\right)$	${\mathit{SOC}}_{\mathit{bat}}$	${\mathit{SOC}}_{\mathit{UC}}$	${\mathit{\eta }}_{\mathit{FC}}$	${\mathit{i}}_{\mathit{bat}}\left(\mathit{A}\right)$	${\mathit{i}}_{\mathit{UC}}\left(\mathit{A}\right)$	${\mathit{P}}_{\mathit{bat}}\left(\mathit{W}\right)$	${\mathit{P}}_{\mathit{UC}}\left(\mathit{W}\right)$	${\mathit{P}}_{\mathit{FC}}\left(\mathit{W}\right)$
1	0.261	4708.9	0.03%	0.65%	11.3%	0.007	0.11	2778.9	1618.7	458.1
2	0.273	5818.7	0.02%	0.68%	13.1%	0.007	0.13	3476.6	2012.5	553.9
3	0.293	5634.5	0.01%	0.61%	11.4%	0.006	0.11	3456.5	1861.9	480.5
4	0.281	5641.1	0.02%	0.32%	14.3%	0.003	0.14	3318.3	1912.1	522
Average	0.277	5450.8	0.02%	0.56%	12.5%	0.006	0.13	3257.6	1851.3	503.7

was made to gain greater insights into the prediction method. It shows the mean absolute error (MAE) between the predicted and actual values for 10 different parameters: speed, load power, battery and ultracapacitor SOCs, fuel cell efficiency, battery and ultracapacitor currents and battery, ultracapacitor and fuel cell powers. In general, the MAE is good for the speed, SOCs and currents prediction. In the case of the load and energy sources powers, the mean deviation was relatively high (around 5000 W) because the prediction model is highly sensitive to differences in acceleration values. Analogously, the deviation in the fuel cell efficiency is also relatively high (around 11% to 14%) because it is highly sensitive to small changes in the fuel cell power values. For instance, there is a pair of points $(P_{FC,predicted},P_{FC,label})=(945.7W,0)\Rightarrow ({\eta }_{FC,predicted},{\eta }_{FC,label})=(61.5\%,0)$ where the big difference between the predicted and the actual efficiencies can be clearly seen.

5. Conclusions

In this paper, a DRL-based method for real-time joint power split optimization for a battery/UC/FC HEV was proposed. First, the TA-LTSM was responsible for predicting the future velocity, which was converted to required load power through the mechanical model. The load power was then optimally split between the two motors and also split between the battery/UC/FC by the proposed SAC agent, which makes use of shaped reward functions to accelerate the training process. Compared to traditional rule-based techniques, the proposed method is robust to different initial SOC values and is able to satisfy the system constraints. Moreover, the results show that the usage of two complementary motors greatly increase the efficiency of the system. Finally, the average MAEs of the prediction step are reliable; therefore, the method may be used to plan in advance the HESS power split.

In the future, a better model for the HEV, including auxiliary systems modeling (air conditioning, lights, sound, power-assisted seats, windows) could be designed to compute a more precise load power. Additionally, the velocity forecast method can currently predict velocity only for the next time step. A better forecast method would be able to forecast the velocity for more time steps, allowing the model to better allocate its resources.Finally, more emphasis could be given to the optimization of the fuel cell to increase its efficiency and its usage whenever the SOC of the ultracapacitor falls below the minimum operating value.