Reinforcement Learning-Enabled Electric Vehicle Load Forecasting for Grid Energy Management

Abstract

Electric vehicles are anticipated to be essential components of future energy systems, as they possess the capability to assimilate surplus energy generated by renewable sources. With the increasing popularity of plug-in hybrid electric vehicles (PHEVs), conventional internal combustion engine (ICE)-based vehicles are expected to be gradually phased out, thereby decreasing greenhouse gases and reliance on foreign oil. Intensive research and development efforts across the globe are currently concentrated on developing effective PHEV charging solutions that can efficiently cater to the charging needs of PHEVs, while simultaneously minimizing their detrimental effects on the power infrastructure. Efficient PHEV charging strategies and technologies are necessary to overcome the obstacles presented. Forecasting PHEV charging loads provides a solution by enabling energy delivery to power systems based on anticipated future loads. We have developed a novel approach, utilizing machine learning methods, for accurately forecasting PHEV charging loads at charging stations across three phases of powering (smart, non-cooperative, and cooperative). The proposed Q-learning method outperforms conventional AI techniques, such as recurrent neural and artificial neural networks, in accurately forecasting PHEV loads for various charging scenarios. The findings indicate that the Q-learning method effectively predicts PHEV loads in three scenarios: smart, non-cooperative, and cooperative. Compared to the ANN and RNN models, the forecast precision of the QL model is higher by 31.2% and 40.7%, respectively. The Keras open-source set was utilized to simulate three different approaches and evaluate the efficacy and worth of the suggested Q-learning technique.

1. Introduction

Electric vehicles (EVs) have been mainly integrated into power grids because of their significant benefit over traditional combustion engine vehicles, which includes bolstered energy self-sufficiency and diminished levels of carbon emissions [1]. There have been numerous creative works on EV recharge infrastructure [2,3]. The surging demand for EVs is introducing intricacy to the energy grid system, leading to difficulties in efficient management. One of the significant apprehensions for executives in this world is accurately projecting the charging capacity of automobiles, to determine their energy consumption requirements [4,5]. Recently, plug-in hybrid electric vehicle (PHEV) energy control methods have used reinforcement learning (RL) [6]. The incorporation of forecasting methods was also used in [7] to explore the merging of PHEVs into microgrids. Furthermore, Ref. [8] considered blockchain technology for trading energy. Additionally, Ref. [9] proposed PHEV energy management with data-driven techniques to estimate battery discharge depth.

The forecasting of EV charging station load has drawn a lot of interest in the literature. For instance, in [10], the load demand for Ev was predicted using online ride-hailing services. To model the load demands of PHEVs and EVs, the authors examined the stochastic load demand of EVs [11]. A statistical utilization strategy has been examined to forecast the PHEVs load profile. The energy shift at midday and night has been predicted using the inverse load flow and a localized charging time-shift procedure [12]. For example, Refs. [13,14] used charging stations with battery storage systems to supplement energy or capacity or to delay demand. Intelligent PHEV charging demand forecasting in [15] aims to keep a constant daily network load profile while ensuring that each PHEVs load requirement is fulfilled in [16]. Ensemble learning predicts load consumption of cooperative PHEVs using innovative data mining to forecast charging sites [17].

1.1. Literature Survey

The battery charging of PHEVs is primarily done at night because everyday vehicle travel considerably lowers the charging time. Due to the high number of these vehicles, early night charging of vehicles damages power infrastructure traits like overloading and fast ramping of energy sources. Increased EV adoption may also lead to more severe grid features like feeder overcrowding, unfavourable peak demand, increased power loss, decreased load factor, harmonic distortion, phase unbalance, etc. The charging of PHEVs during high demand has these negative impacts [18]. Smart recharge for electric vehicles is the remedy to the problems mentioned above [19]. EVSC can effectively handle EV charging, particularly at night, to satisfy the network’s technical restrictions. According to data, vehicles are typically only driven 4 to 5 percent of the time, with the remainder of the day being spent in parking lots or basements [20]. The extra storage space provided by the batteries of electric vehicles can be used to create a large-scale energy storage system [21].

PHEV owners have expressed high satisfaction with the cooperative charging of EVs, and grid characteristics are considered when deciding the number of EVs to be charged per time interval [22]. In addition to considering the use of renewable energy production in a power network, the EVSC process should also consider PHEV owners and technological limitations on renewable-based power [23]. PHEVs have the potential to contribute significantly to helping keep or enhance the normal functioning of a power grid. PHEV charging via EVSC can also provide extra services like frequency regulation [24]. The schematic diagram of EVSC is shown in Figure 1. Both centralized and decentralized types of charging—in which electric vehicles are refuelled in intelligent houses, at home or by fleet operators, as well as parking lots occupied by EVs parked overnight—can be used to refill the batteries of electric cars. EVSCs flexible charging and auxiliary services can reduce the energy cost of plug-in cars by as much as 60% when compared to conventional vehicles [25].

A company that operates a fleet of electric vehicles must be able to forecast its load in order to optimize the number of EVs, their charging rates and available charging sites. Additionally, more energy-efficient use of existing resources can be enabled by predicting the future capacity needs [26,27,28]. The load profile of EVs in particular geographic region results from the habits people have while driving and contributes to the overall demand for electricity [29,30]. A method has been developed to forecast electric vehicle travel patterns and arrival/departure times. In addition, an ensemble learning-based forecasting method can predict the number of charging stations required in any given area [31,32]. The proposed method was developed by combining three different types of learning algorithms: Linear regression (to find the weighting for each primary learner), recurrent neural networks, and long short-term memory. The ability of three neural network models to predict the load profile in an EVS has been examined in other research [33]. The Radial Basis Function technique has produced better results in load forecasting than the traditional method, but it also had a higher error rate and processing cost. In [34], six distinct deep-learning techniques have been examined from the EV usage prediction perspective. The methods included canonical LSTM, ANN, RNN, gated recurrent units, bi-directional LSTM forecasting algorithms, and stacked auto-encoders.

1.2. Motivation and Problem Statement

Deep learning (DL) techniques are increasingly being used to improve the prediction performance of highly nonlinear systems to reduce the computational burden on a design and allow the real-time application of complex controllers [34]. The literature achieves the aforementioned goals by utilizing a variety of machine learning (ML) methods [35]. Artificial neural networks (ANN) were mainly used in [36] for datasets with no time dependency between the accessible datasets. Time-varying data sets are commonly processed using recurrent neural networks (RNNs).RNNs and CNNs are two popular AI techniques used to gather data for text and media applications, respectively [12]. SARSA and Q-learning are two additional reinforcement learning techniques that facilitate real-time learning of intricate problems. The action-reward system underlies these techniques [32,33,34,35,36,37]. Consequently, the RL technique may impose a different reward for each activity that moves an agent from one condition to another. This paper foresees PHEV loads using one of the RL techniques, the Q-learning (QL) approach. Out of all possible actions and states, the QL method can identify the best ones. As a result, by using the outputs of other ML techniques, like ANN, LSTM, RNN, GRU, and CNN, as initial inputs, the efficacy of the QL technique can be improved [38].

Despite the variety of methods used to predict PHEV load demand [11,12,13], more research still needs to be focussed on an overarching tactic that encompasses all PHEV load demand scenarios, including smart, cooperative, and non-cooperative situations. In this study, the authors have explored the potential of an RL-based QL approach in forecasting PHEV load under diverse scenarios. This research evaluated three commonly used artificial intelligence techniques–RNN, ANN, and QL–to determine their efficacy in prediction. The investigation indicates that implementation of the QL technique, which leverages the initial outcomes of the ANN and RNN methodologies, yields enhanced predictive precision.

1.3. Real Contribution

The leading contributions and innovations of this article are as follows:

• A comprehensive investigation of load forecasting challenges for PHEV charging remains imperative through implementing RL, a potent tool for combining multiple ML models [6]. To address this matter, a new approach based on QL for forecasting load at EV charging stations has been introduced in this publication. QL has been frequently employed in problems with distinct states and actions. Our proposed QL model is suitable for the control task that demands ongoing response to the prevailing circumstances. The system comprises neural networks and deduces the differential for the state evolution of unknown epistemic uncertainty. This solution presents an opportunity to enhance the operational efficiency of PHEV charging, while also serving as a mechanism of reinforcement for energy dispatching within power grids.
• The recommended RL methodology for determining the optimal framework for PHEV load forecasting encompasses smart, cooperative, and non-cooperative scenarios. The developed QL approach exhibits superior efficiency, precision, and flexibility in PHEV load estimation when compared to traditional ANN and RNN models. Furthermore, incorporating modifications such as adjusting the epoch, hidden layer, and node quantities can significantly augment the accuracy of PHEV charging load predictions, as evidenced by empirical analyses.

1.4. Paper Organization

This article is organized as follows. Section 2 presents the technical background for the three models utilized in this paper. Section 3 describes the charging behaviour of the PHEVS. Section 4 presents the proposed framework for the PHEVs load forecasting. Section 5 and Section 6 show the evaluation criteria and QL-model forecasting performance with different network depths. The implementation of ANN, RNN and the proposed model and simulation results considering the test cases are described in Section 7. Sensitivity analysis, validation of the proposed model, QL in terms of speed, flexibility and accuracy, and discussion are presented in Section 8, Section 9, Section 10 and Section 11, respectively. Finally, Section 12 concludes the article.

2. Technical Background

The three models—ANN, RNN, and QL—used in this paper are briefly introduced in this part.

2.1. ANN

Figure 2a shows the schematic unit of ANN [23]. The Artificial Neural Network (ANN) comprises three distinctive layers i.e., input layer (X), hidden layer (H), and output layer (Y) that operate using multiple nodes. As shown in Figure 2a, a weight (${\omega }_{ij}$) is used in every line of the ANN technique in between each pair of successive layers (ith previous layer, and jth present layer). The Equation (1) can be used to determine each output node:

$$N{O}_j^p={\sigma }_j^p\left(\sum _{i=0}^{n_{p-1}}\left(N{O}_i^{p-1}{\omega }_{ij}\right)+b_j^p\right)$$

(1)

In the context of neural networks, the jth output of the pth layer is represented by $N{O}_j^p$, whereas $N{O}_j^{p-1}$ represents the jth output of the $p-1$th layer. Similarly, $b_j^p$ and ${\sigma }_j^p$ respectively refer to the bias and activation function of the jth node in the pth layer. Furthermore, $n_{p-1}$ denotes the number of nodes in the $p-1$th layer.

2.2. RNN

The RNN technique’s block layout is depicted in Figure 2b. The time-series data set’s future is predicted using RNNs [24]. It can be applied to various chores, including predicting weather patterns, green energy trends, stock market trends, and other activities. The line the RNN method uses between each hidden layer’s node is the primary distinction between it and the ANN technique. The output of the RNN unit is shown in Equation (2):

$$\begin{array}{c}\left\{\begin{array}{c}{r}_t=b+K{s}_{t-1}+W{x}_t\hfill \\ s_t=f\left(r_t\right)\hfill \\ q_t=g(d+U{s}_t)\hfill \end{array}\right..\end{array}$$

(2)

At time t, the state of the hidden layers is denoted by $s_t$, while at time $t-1$, it is indicated by $s_{t-1}$. Similarly, the input and output of the RNN unit at time t are represented by $r_t$ and $q_t$, correspondingly. K, W and U are the weight matrices of $s_{t-1}$, $x_t$, and $q_t$ respectively. Both f and g are the activation functions (softmax, tanh, etc.) of memory and output of the RNN unit, while b and d are the biases.

2.3. QL

The QL method, which uses the Markov decision process to select the most appropriate action from all possible options, is one of the model-free RL techniques [25,26]. QL falls under the category of off-policy methods. When choosing the best future action, the QL technique selects the initial state, reward, current state, and accessible activities in a very similar way to the SARSA. The primary equation, which is used by the QL technique to choose, presented in (3):

$$\begin{array}{c}\left\{\begin{array}{c}{Q}^{\mathrm{new}}(c_t,m_t)\leftarrow Q(c_t,m_t)+\beta (R_t\hfill \\ +d_f\underset{m}{max}Q(c_{t+1},m_t)-Q(c_t,m_t))\hfill \end{array}\right.\end{array}$$

(3)

where $c_t$ and $m_t$ are the current states and current actions of the QL technique. $Q(c_t,m_t)$ is the Q-value updated Q-value of $c_t$ and $m_t$ can be denoted as $Q^{\mathrm{new}}(c_t,m_t)$. The QL method has a learning rate $\beta$ that satisfies the condition $0\le \beta \le 1$, while the discount factor $d_f$ is between 0 and 1 ($0\le d_f\le 1$). The optimal future value estimation is given by ${max}_m\left(Q(c_{t1},m_t)\right)$. The proposed QL reward for the future action and state is $R_t$. The initial conditions for the Q-value in this proposed technique are zero. The optimal solution for PHEV loads is found using the QL method. The proposed QL technique updates the Q-value depending on $c_t$ and $m_t$ after each iteration. The best course of action for the subsequent steps is determined using Equation (4):

$$\begin{array}{c}{m}_t=arg\underset{m}{max}\left(Q(c_{t+1},m_t)\right).\end{array}$$

(4)

3. Charging Behavior of PHEVs

The charging strategy, charging time, amount of charged PHEVs, battery capacity, battery state of charge ($B_{soc}$), and start time are all unknown variables that affect PHEV load behaviour. The three charging cases for PHEVs—smart, cooperative, and non-cooperative—are explored in this article.

3.1. Non-Cooperative PHEV Charging

The simplest charging approach for PHEVs is the non-cooperative strategy, which involves plugging them into charging stations. PHEVs are believed to commute daily from their homes in the morning to their homes in the evening. In this instance, around 5:30 p.m., when most people get home, most PHEVs are hooked up and begin charging. As a result, this strategy considers a limited range uniformly distributed probability density function (PDF) with a charging start time of 5:30 p.m. Such a PDFs mathematical form is provided in Equation (5) [25]:

$$y\left({\tau }_x\right)=\frac{1}{u-v},v=18,u=19,u\le {\tau }_x\le v$$

(5)

where u and v are constant values, ${\tau }_x$ is the charging start time of PHEV.

3.2. Cooperative PHEV Charging

Stockholders in the cooperative charging plan typically link their cars to chargers during off-peak hours to prevent confrontation with the sunset peak hours, when power costs are higher. Consequently, to save money on their electricity prices, people defer charging until after 9:30 p.m. The cooperative charging strategy presented in Equation (6) is then modelled using the provided pdf:

$$y\left({\tau }_x\right)=\frac{1}{u-v},v=21,u=24,u\le {\tau }_x\le v$$

(6)

3.3. Smart PHEV Charging

A smart charging plan schedules charging when energy is most reasonably priced, demand is at its lowest point, or excess capacity exists. All smart charging techniques adhere to the fundamental principle that a vehicle should only be charged when doing so benefits both the owner and the utility. A typical pdf, as shown in Equation (7), illustrates the difficulty of choosing the charging start time using various smart charging options [24]:

$$y\left({\tau }_{\mathrm{x}}\right)=\frac{1}{\beta \sqrt{2\pi }}ex{p}^{\left(-\frac{1}{2}{\left(\frac{{\tau }_{\mathrm{start}}-\alpha }{\beta }\right)}^2\right)},\alpha =1,\beta =3.$$

(7)

where $\alpha$ is the mean of values and $\beta$ is the covariance.

Once the PHEV is connected to the home charger, the battery begins to charge. The battery’s leftover $B_{soc}$ is calculated using the vehicle’s daily mileage. The ratio of available energy to maximal stored energy is known as a battery’s $B_{soc}$. The daily mileage of a car is said to follow a log-normal PDF presented in Equation (8):

$$y\left(z\right)=\frac{1}{z\beta \sqrt{2\pi }}ex{p}^{\frac{-{(ln(z)-\alpha )}^2}{2{\beta }^2}},z>0.$$

(8)

The $B_{soc}$ at the time of plug-in is then calculated using the vehicle’s driven mile and all-electric range ($A{E}_{range}$) as follows in Equation (9):

$$B_{soc}=\left\{\begin{array}{cc}0,\hfill & \hfill z>A{E}_{range}\\ \hfill \frac{A{E}_{range}-z}{A{E}_{range}}\times 100\%,& \hfill z\le A{E}_{range}.\end{array}\right.$$

(9)

A multitude of PHEVs exist, including the PHEV-20, PHEV-30, PHEV-40, and PHEV-60, each characterized by a unique $A{E}_{range}$ denoted by a numerical subscript. The PHEV-20 has been selected as a primary example in the investigation of market potential over time [29], but consideration may be extended to additional PHEV models given the unrestricted nature of the methodology. Equation (10) is used to determine the charging time of a PHEV [29].

$${\tau }_d=\frac{B_{cap}\times (1-B_{soc})\times D_{depth}}{\eta \times T}.$$

(10)

where $B_{cap}$, $D_{depth}$, $\eta$ and T and the PHEV battery capacity (in kWh), depth of discharge of PHEV battery, charging efficiency, and PHEV charger rate (kW), respectively. The charging types for PHEVs are outlined in the

Table 1. Various Charger Types for PHEVs.

Charging Type	I/P Voltage	${\mathit{P}}_{\mathit{max}}$ (KW)
Level-I (AC)	120 $V_{ac}$	1.42
Level-II (AC)	208–240 $V_{ac}$	11.5
Level-III (AC)	208–240 $V_{ac}$	97
Level-III (DC)	208–600 $V_{dc}$	239

, and are contingent on the charging levels provided by the charger [20]. Ethics dictate that the first two charging stages are suitable for PHEVs that are charged domestically. As the charging stations with the third stage are specifically constructed for public transportation, this write-up excludes their inclusion. As per

Table 2. Various Classes for PHEVs.

Class	Market Share	${\mathit{B}}_{\mathit{cap}}$ (Min–Max)
Mini Vehicle	0.2	8–12
Mid Size Vehicle	0.3	14–18
Economy Vehicle	0.3	10–14
Light Truck	0.3	19–23

, PHEVs are segregated into four groups, with each group possessing characteristic market and share traits. In this study, the market share of PHEVs can be represented with a discrete distribution. Each PHEV group is arbitrarily selected from the market shares shown in Equations (11) and (12) using a normal distribution.

$${\alpha }_{B_{cap}}={\displaystyle \frac{\mathrm{Min}{B}_{cap}+\mathrm{Max}{B}_{cap}}{2}}$$

(11)

$${\beta }_{B_{cap}}={\displaystyle \frac{\mathrm{Min}{B}_{cap}-\mathrm{Max}{B}_{cap}}{4}}$$

(12)

The proposed QL approach can compile training data to forecast PHEV charging demand using three simulated charging methods: Smart, non-cooperative, and cooperative, based on the (5) to (12). Min and Max represent the minimum and maximum values, respectively.

4. Proposed QL-Based PHEV Charging Load Forecasting Framework

In this article, using the results of RNN and ANN, we developed a novel RL-based model that optimally forecasts the load demands of smart, cooperative, and non-cooperative PHEVs.The proposed model has two major parts: (1) Data Preprocessing Phase, (2) QL-based forecasting considering the optimal results of ANN and RNN as initial inputs. Figure 3 shows the structure of the proposed reinforcement learning-based forecasting model.

4.1. Data Source

Caltech has made the ACN-Data [39] available to academics for research purposes; this dataset comprises EV charging events from various workplaces. More than 30,000 charging events are organized in the dataset from two Californian workplace charging locations, the Jet Propulsion Laboratory, and the Caltech campus. For this article, our test data consists of charging information collected from the charging station on the Caltech campus. The ACN (Adaptive Charging Network) is open to the public and equipped with a 50 kW DC rapid charger, frequently used by non-drivers. The dataset covers details of all transactions made within the architecture of ACN, with 54 EV supply equipment. The dataset’s related data categories are described in

Table 3. Data fields in ACN-Data.

Field	Description
Time of connection	The plugs in time of users.
Accomplished charging time	The time of last non-zero charging rate.
Time of disconnection	The unplugs time of users.
kWh supply	Supplied energy measurement.
Session ID	Unique identity for the session.
Station ID	Unique identity of the EV Supply Equipment.

. The exactitude of load forecasting depends on data pre-processing because the initial data will contain anomalies and missing numbers.

4.2. Preprocessing Module

The EV charging stack data were initially analyzed and refined for inquiry. To enable real-time charging control based on energy usage during charging periods, it is imperative to establish the average hourly charging capacity of the stations, taking into account the variability of data encountered due to its random nature. To achieve this, the dataset was pre-processed, whereby inaccurate data points were replaced with the average charging load observed during a given time slot across different days. This step was essential to ensure accurate forecasting without being impacted by erroneous data.

For practical reasons, the data pertaining to the charging load operation were bifurcated into training and test sets using a proportion of 0.7/0.3. The model was suitably trained with the training instances, while the final function evaluation utilized the test collection. Multiple neural networks were employed to learn from the charging load data; however, the efficiency of these networks was significantly influenced by the size of the data. As a resolution, the input data of the forecast model were altered through min-max normalisation, which constrained the information within a specific limit (0–1), as illustrated in Equation (13):

$$Y^{\prime }=\frac{Y-y_{min}}{y_{max}-y_{min}}$$

(13)

4.3. QL-Based Forecasting Module

The three primary learners, ANN, RNN, and QL, were trained using the training set by the training technique. Following the last training session, predictions were made using the training data and prepared elementary learners. Forecasts from the training data and the proper labels were then set as the features. During the testing process, predictions were generated using the trained base of learners on test data. The forecasts made by the primary learners were then used to test the learned weighted averaging model. After reversing normalization, the outcomes were obtained, and the test labels and marks used to calculate the prediction accuracy.

• When using the ANN technique to forecast the load on PHEVs, the input and output ANN units should be chosen appropriately. Due to the time series nature of PHEVS load data, the ANN unit utilized prior PHEV load data. The baseline 24-h PHEVs load data were helpful for more accurate one-hour-ahead load forecasting. The deployment of ANN and RNN for predicting one hour ahead is shown in Figure 4.
• The proposed QL method for the PHEVs load forecasting used the previous days’ ANN and RNN forecasting results. In hopes of identifying the best day-ahead PHEV load forecasting, the proposed QL approach chose the best course of action based on the output of ANN and RNN. The proposed QL model’s reward function is shown in Equation (14):

  $$\begin{array}{c}{R}_t\left({\gamma }_j\right)=\frac{1}{exp\left|(J_{Optima{l}_t}-{\gamma }_j(J_{ANN}/RN{N}_t))\right|}\end{array}$$

  (14)

  where ${\gamma }_j$ is the random number of the jth action. The proposed approach chose one of two possible courses of action for each time horizon ($j=0$ for ANN and $j=1$ for RNN). The power forecast for the ANN or RNN method was $\left(J_{ANN/{RNN}_t}\right)$. Additionally, $J_{{Optimal}_t}$ contains the PHEV data for the tth period. The ANN and RNN predicted results were used in the devised method to predict the PHEV future load and compare it with actual data. Thus, 24 optimal actions were chosen in the proposed QL approach for the day ahead horizon. Therefore t can be any integer between 1 and 24 for the day ahead horizon. $J_{Optima{l}_t}$, $J_{AN{N}_t}$, and $J_{RN{N}_t}$ were the optimal, ANN, and RNN PHEV load of the tth day, respectively. The ${\gamma }_j$ values between 0 and 2 were used to improve the ability of the QL technique to find the best search locations. When the action was 1, the RNN technique chose the reward function. Alternatively, the reward function was determined from the results of the ANN method.

QL is a learning system that is built on values. Value-based algorithms update the value function based on a calculation (particularly Bellman equation). The other policy-based version calculates the value function using the most recent policy improvement’s greedy policy. QL is an ad hoc student. This implies that it learns the value of the best strategy independent of the agent’s behaviours. On the other hand, an on-policy learner learns the value of the policy being carried out by the agent, including the exploration stages, and it will find an optimum strategy that considers the exploration implicit in the approach.

5. Evaluation Criteria

The predicting performance of the analyzed techniques was evaluated using the mean square error (MSE), a standard evaluation criterion. Considering $P^{test}$ and ${\widehat{P}}^{test}$ are the actual and expected loads of the EV charging station at time step t, respectively. The MSE measured the errors between the real and the forecasting values, which can be formulated in Equation (15) as below:

$$MSE=\sqrt{\frac{{\sum }_{t=1}^T{\left(P_t^{\mathrm{test}}-{\widehat{P}}_t^{\mathrm{test}}\right)}^2}{T}}$$

(15)

6. QL-Model Forecasting Performance with Different Network Depths

We evaluated the predicting effectiveness in comparison to various network levels. The results of the Mean Squared Error (MSE) and Mean Absolute Error (MAE) that were found when the layers were changed from two to six when using the prepared dataset are shown in Figure 5. The proposed QL model worked best with five layers. As the network depth increased from two to five, the MSE and MAE values became smaller, and the prediction performance improved. Properly boosting network depth and settings significantly enhanced the predicting performance. Due to increased parameter redundancy and reduced data variety, as a network’s depth grows, it may overfit and suffer from performance loss. The training period needed for each layer of QL is listed in

Table 4. Training time for different layers of QL.

Layer of QL	CPU Time (s)
2	499.48
3	798.21
4	1401.98
5	2001.73
6	2312.19

. As layers increased, factors for training optimization increased, causing a longer required time. More layers were needed to improve forecast accuracy, resulting in time consumption.

Convergence of the QL

In Figure 6, the state $(c,m)=(0,50)$ marks the convergence of the QL model. The optimal value function $V(0,50)=160.83$ is approached as ${min}_{m\in \mathcal{A}}Q(0,50,m)$ converges. The synchronized parallel Q-learning procedure involves offline learning of the QL model, in which batch updates of all state-action pairs (Q-functions) occur at each training instance. The step sizes utilized in this process are inversely proportional to the number of visits (updates) to each combination of state and action.

7. Test Cases Simulations and Results

This section compares three techniques (RNN, ANN, and QL) in various scenarios to show the benefits and efficacy of the proposed ML techniques. The number of hidden layers used for RNN and ANN is shown in

Table 5. Examining and projecting the charging demand of PHEVs utilizing ANN, RNN, and QL approaches (MSE, MAPE and Epoch) across various penetrations (30%, 60% and 90%).

Techniques	Charging Strategy	Penetration (%)	MSE (KW)	Epoch	MAPE (%)
ANN-1 (non-cop)	Non-cooperative	30	4.2	3000	4.3371
RNN-1 (non-cop)			9.3	1000	2.9189
QL-1 (non-cop)			0.79	10,000	2.4741
ANN-1 (Cop)	Cooperative	30	7.06	3000	4.4901
RNN-1 (Cop)			9.12	500	3.0129
QL-1 (Cop)			6.21	10,000	2.7210
ANN-1 (Smart)	Smart	30	6.23	3000	4.3210
RNN-1 (Smart)			6.38	500	2.7112
QL-1 (Smart)			5.30	10,000	2.4214
ANN-2 (non-cop)	Non-cooperative	60	1.67	3000	5.2121
RNN-2 (non-cop)			25.63	500	3.1489
QL-2 (non-cop)			1.30	10,000	2.9741
ANN-2 (Cop)	Cooperative	60	9.54	3000	5.7371
RNN-2 (Cop)			9.67	500	4.2189
QL-2 (Cop)			9.37	10,000	3.1451
ANN-2 (Smart)	Smart	60	7.12	3000	5.9871
RNN-2 (Smart)			7.34	500	4.0189
QL-2 (Smart)			5.23	10,000	2.9741
ANN-3 (non-cop)	Non-cooperative	90	0.0031	1000	3.2171
RNN-3 (non-cop)			0.0019	1000	2.7189
QL-3 (non-cop)			0.0019	1000	2.2741
ANN-3 (Cop)	Cooperative	90	1.23	3000	3.3371
RNN-3 (Cop)			10.3	5000	2.8189
QL-3 (Cop)			0.889	10,000	2.1741
ANN-3 (Smart)	Smart	90	7.11	3000	3.6371
RNN-3 (Smart)			7.45	5000	2.7189
QL-3 (Smart)			4.45	10,000	2.0741

. The number 24 is the input for the ML methods. (For example, the charging load for the following hour is forecast using the most current 24-h data). This analysis estimated the devised PHEV load forecasting approach using the free and open-source Keras software [26]. Additionally, MATLAB was utilized to build the original PHEV data for various charging techniques.

7.1. Load Forecasting of Non-Cooperative PHEVs Charging

This article predicts non-cooperative PHEV load demands using ML approaches such as RNN, ANN, and the proposed QL techniques. A predicted load with a 30% PHEV adoption is shown in Figure 7a. The necessary information is given in Table 5, along with the epoch numbers and means square errors (MSEs) used for the different ML techniques in this work during the training phase. ANN and QL methods have lower MSEs but higher accuracies than the RNN strategy. The more significant epoch number used during training was the primary factor that makes the ANN technique perform better in terms of accuracy than the RNN method. The non-cooperative PHEV hourly load forecasting with 60% penetration is shown in Figure 7b. The charging of the PHEV load started at 5:30 p.m. and ended at 5:30 a.m. depicted in Figure 7. Additionally, the AI technique had a 60% penetration of PHEV load and an insignificant MSE (significant accuracy). Table 5 shows that the RNN method had a more significant mean squared error (MSE) and smaller epoch number than other techniques. The devised QL method predicted PHEV load using ANN or RNN outcomes with lower MSEs than the other two methods, which makes this approach more accurate. This illustration shows how the QL approach was more valuable than ANN and RNN for predicting PHEV loads.

7.2. Load Forecasting of Cooperative PHEVs Charging

The predicted load consumption for cooperative PHEVs is given in this subsection. When PHEV penetration is 30%, Figure 8a shows the cooperative PHEVs hourly load forecasting with 30% penetration using ANN, RNN, and QL algorithms. Training iterations (1000 (ANN), 3000 (RNN), and 10,000 (QL)) were used. The cooperative PHEV charging strategy lasted from 7:00 p.m. to 11:00 a.m. as shown in Figure 8. As shown in Table 5, the QL method predicted the cooperative PHEVs hourly charging load more precisely (with a reduced MSE) than the ANN and RNN approaches. Additionally, the output of the QL method in Figure 8 demonstrates less deviation from the actual data.

7.3. Load Forecasting of Smart PHEVs Charging

The 60% and 30% penetration levels of the smart PHEVs charging hourly load are forecasted in this subsection. The training iterations (1000 (ANN), 3000 (RNN), and 10,000 (QL)) are used. Figure 9 shows that all techniques were capable of forecasting PHEV loads with accuracy. However, because the RNN method employed fewer iterations than the ANN and QL techniques, its error in calculating the PHEV load was more important. The proposed QL approach has a reduced MSE (a better precision) than the ANN and RNN methods because it chose the optimal policy from the potential actions for each state (each hour). The proposed strategy’s accuracy can be improved by applying the QL method to more extensive and diverse data sets.

8. Sensitivity Analysis of Three Charging Techniques

Figure 10 shows the sensitivity evaluations of three PHEV charging load strategies implemented with different hidden layers, node numbers, and several neurons in the proposed QL. Figure 10a shows how the proposed QL method’s forecasting accuracy varied with different epoch spans when increased from three to four hidden layers, each having 200–400 nodes. Table 5 shows that when more nodes and hidden layers were used in the ML model, it accurately predicted PHEV loads for the cooperative charging method. Table 5 shows that the quality of a forecast was improved by increasing the number of nodes in each hidden layer. It can be seen from the Figure 10b that the QL technique for the non-cooperative PHEV load forecasting was unable to model the PHEV load filling consistently. The more layers and nodes a QL has, the better it can handle large datasets. Figure 10c shows the sensitivity analysis of the QL method for smart PHEV load forecasting with various nodes and hidden layers when penetration was 30%. Therefore, it can accurately predict PHEV load if the proposed QL method employed more nodes and hidden layers than other techniques—such as ANNs or RNNs.

9. Validation of Proposed QL

To produce accurate forecasting results for comparison, the parameters and hyperparameters of the QL approach should be developed appropriately. The validation test selected these values through trial and error. For instance, the MAPE evaluated QL efficiency, as seen in Figure 11. The QL with five hidden layers among ten guaranteed the best MAPE and the QL selected 41 hidden neurons among fifty to achieve the smallest MAPE.

10. QL in Terms of Speed, Flexibility and Accuracy

The flexibility, speed, and accuracy of the three DL methods are discussed with their ability to predict PHEV charging loads.

10.1. Faster Speed

In this paper, we integrate two ML methods—ANN and RNN—to enhance the QL method’s real-time ability to forecast PHEV load. Our three ML methods focus on training times but vary substantially across applications. When ANN and RNN have trained accurately with PHEV data, their networks can efficiently predict the input data to implement the QL technique in real-time. Therefore, utilizing ANN and RNN techniques to their full potential can significantly improve the QL technique’s capability to carry out this process quickly.

10.2. Improved Accuracy

It is essential to examine all DL techniques currently being used; after evaluating the three ML techniques described in this article, the most efficient QL for forecasting PHEV charging loads was chosen. From the results of the simulations, predictions from ML techniques may be inaccurate if there are substantial variations from the initial charging points. Therefore, the QL approach, which gets its data from ANN and RNN, may precisely forecast optimal PHEV charging. As Table 5 demonstrates, increasing the epoch number can reduce forecasting errors.

10.3. Flexibility

The simulation results show that ML techniques can be used to model PHEVs by training charge methodologies. Any complex load-charging scenario can be variably predicted by increasing the training iterations for ML techniques. The estimate for PHEV charging potential was correct as a consequence. Forecasting PHEV loads for cooperative, non-cooperative, and smart charging can be elevated by combining nodes and hidden layers of ANN and RNN techniques. Moreover, the QL technique’s capacity to choose the best course of action out of numerous choices can be improved by having more precise estimates of PHEV load charging.

11. Discussion

A significant study is being done to increase the accuracy of load forecasting for PHEV charging sites, which can both guide the efficient dispatch of the power system and advance the growth of EVs. This study proposes a QL method for forecasting the demand at electric car charging sites. The data preprocessing unit handles the original dataset, builds time series data, and adds characteristics to prediction data in our technique. We use the QL forecasting technique and approximate the posterior distribution using variational inference. Results show the success of this method in predicting PHEV charging station demand. Comparing the devised method to some forecast methods further indicates its better performance. The designed model is around 38.2%, 41.5% lower in MSE and MAE, and 17.9% higher than the standard techniques, according to the findings of point forecasting. The devised strategy can be used to forecast load for PHEV charging stations in the real world, as evidenced by its outstanding performance on various datasets.

12. Conclusions

This manuscript presents a novel QL approach for PHEV load forecasting employing ANN and RNN techniques. Various PHEV models, such as smart, cooperative, and non-cooperative, are analyzed in this study. Our results indicate that the proposed QL method achieved accurate predictions of PHEV load charging by leveraging data from the ANN and RNN techniques. Importantly, our findings suggest that the QL method outperformed the ANN and RNN techniques in predicting PHEV loads accurately, as substantiated by the simulation outcomes. Table 5 shows that in the worst-case scenario for PHEV charging (smart charging), the QL method exhibited superior performance compared to traditional ANN and RNN techniques, surpassing them by over 60%. Furthermore, increasing the number of iterations demonstrated that the ANN technique delivers more precise predictions of PHEV loads (MSE) than the RNN approach. The QL methodology under consideration demonstrates superior tracking capabilities for PHEV loads when compared to ANN and RNN techniques, exhibiting greater precision and adaptability in the process. Further enhancements in the form of adjustments to the hidden layer, epoch, and node numbers have been shown to significantly enhance the accuracy of PHEV charging load forecasts, as corroborated by the case studies.The forecasting methodology developed here demonstrates the potential for future implementation in expansive power networks that contain complex PHEV loads. The performance of this probabilistic forecasting technique can be improved through ongoing optimization efforts. Reliable load projections derived from PHEV charging facilities will facilitate a structured approach to electric vehicle charging, and effectively reduce energy usage variations while optimizing distribution network resource allocation. We are committed to further forecasting the load from various charging stations, including those with AC, DC, and higher power capabilities. We plan to extend our suggested model to more PHEV charging data sets. Additionally, we intend to utilize this methodology in additional applications, including dispersed grid capacity forecasting and wind power forecasting, to expand its potential impact.