A New Deep Learning Restricted Boltzmann Machine for Energy Consumption Forecasting

Abstract

A key issue in the desired operation and development of power networks is the knowledge of load growth and electricity demand in the coming years. Mid-term load forecasting (MTLF) has an important rule in planning and optimal use of power systems. However, MTLF is a complicated problem, and a lot of uncertain factors and variables disturb the load consumption pattern. This paper presents a practical approach for MTLF. A new deep learning restricted Boltzmann machine (RBM) is proposed for modelling and forecasting energy consumption. The contrastive divergence algorithm is presented for tuning the parameters. All parameters of RBMs, the number of input variables, the type of inputs, and also the layer and neuron numbers are optimized. A statistical approach is suggested to determine the effective input variables. In addition to the climate variables, such as temperature and humidity, the effects of other variables such as economic factors are also investigated. Finally, using simulated and real-world data examples, it is shown that for one year ahead, the mean absolute percentage error (MAPE) for the load peak is less than 5%. Moreover, for the 24-h pattern forecasting, the mean of MAPE for all days is less than 5%.

1. Introduction

The most effective usage of power plants depends on many elements, such as weather variables like wind, humidity, and clouds, as well as other considerations such as holidays, months of the year, and days of the week. Due to its remarkable relevance for system operation and development, load forecasting has drawn considerable attention [1,2].

Proper load forecasting reduces investment costs and enables better planning for the construction of distribution and transmission networks. Electric loads are a dynamic, ever-changing characteristic. As a result, planning should be performed while considering reliability coefficients and the maximum load. Until future development, the planned network must also satisfy the demands of the area [3,4].

The load forecasting is classified into three levels: short-term, mid-term, and long-term. The applications and the methods of the three levels are different. At the short-term level, the forecasting horizon is just a few days, and it is used for short-term planning such as optimal use of power networks. Mid-term load forecasting (MTLF) denotes a monthly or, at most, annual forecast and is often used to manage peak consumption in certain seasons. The long-term level denotes the forecasting for some years ahead and is used for long-term planning, such as managing new power plants [5,6].

The remainder of this paper is organized as follows. Many different approaches for time series analysis and forecasting are reviewed in Section 2, and the main contributions of this study are listed at the end of this section. The effective variables are analysed in Section 3. In Section 4, the suggested RMB and the learning machine are described. The 24-h pattern forecasting and the forecasting of weekdays and weekends are studied in Section 5. In Section 6, the suggested application is described. Comparisons between MAPE and two different ANN-based methods are made in Section 7. Finally, the conclusions are given in Section 8.

2. Literature Review

MTLF helps to improve congestion planning in transmission networks, thus enhancing the system’s overall efficiency and optimizing energy costs for the consumer [7,8]. The advantage of exact MTLF is that companies can use it to improve their transmission network and distribution system. MTLF has been widely studied. In [9] investigated the economic impact of MTLF in the last two decades in market regulation and improvement of the transmission network and distribution system. Today, MTLF is also considered in energy transactions and assists in purchasing or selling energy and developing generation, transmission, and distribution contracts on a monthly or annual basis. It also affects the contractors of generators and distributors. Incorrect forecasts may lead to insufficient supply or oversupply. Therefore, it can be realized that exact MTLF leads to a more economical system [10].

The impact of climate conditions on the mid-term horizon has been comprehensively investigated in [11]. In [12], an MTLF scheme using the autonomous modelling technique is suggested to predict the monthly load, where the effective variables are only loaded and climatic variables. In this study, the authors compare the performance of the statistical method with the results of artificial intelligence and conclude that the method based on artificial intelligence yields better results. It has also been shown that statistical preprocessing is required for data analysis. In [13,14], an artificial neural network (ANN) approach is suggested that offers better results than statistical methods (regression models). In this reference, the forecasting of monthly load demand for one year is examined. In [15], an MTLF model based on a dynamic ANN is proposed. The model presented in this reference is compared with the statistical approach, and it is concluded that the values forecast by the proposed method are more accurate. The main advantage of the model proposed in this paper is that meteorological forecasts are not used. Climate variables rarely have accurate forecasts on a horizon of more than a week. In [16], an MTLF model based on the neural network approach is presented without any climatic information. The transformer models are suggested in [17,18] for load forecasting.

The deep learning methods, fuzzy systems and ANNs are widely used for modelling and forecasting problems [19,20,21]. In a few studies, these methods have also been applied to load forecasting. For example, a short-term forecasting scheme is designed in [22], and the effect of weather conditions is studied. In [23], the optimized ANNs by particle swarm optimization approach is suggested for MTLF, and by various examinations, the reduction of the number of input variables is studied. In [24], an ANN approach is developed, and the elimination of climate variables from the list of input factors is investigated. In addition, the effect of population growth is investigated by the designed ANN model. In [25], the days of the week are divided into three sections: early days of the week, middle days of the week and the weekend, and a separate model is proposed for each section. In [26], the days of the year are divided into 12 parts, and a neural network model is presented for each part. In some sources, the problem data are divided into winter and summer. In [27], genetic algorithm-optimized neural networks are used for medium-term forecasting. In [28], radial neural networks are used for modelling and forecasting. In the method proposed in this reference, neural network inputs include time (year, month, number of days of the month and days of the week), economic factors and temperature. The economic factor and temperature forecast must be available to forecast load at a specific time. Ghaderpour et al. [29,30] utilized the least-squares spectral and wavelet methods to approximate the seasonal cycles in the geodetic, climate and streamflow time series that can be used for forecasting. In [31], a comprehensive review is presented to inveigle the various models for energy consumption.

In most of the MTLF methods above, (1) the specific region is investigated, and the presented approach cannot be easily applied to other regions; (2) in some described MTLF methods, the computations are too heavy and cannot be applied to real-world problems; and (3) in the ANN-based methods, conventional learning schemes are used. Regarding the discussion above, a new deep learning approach is suggested for MTLF. The main contributions include:

• A comprehensive analysis is presented to evaluate the effect of various factors, such as economics, climate and load pattern.
• A deep learning approach based on RBMs and CD algorithm is presented that can be easily applied to a real-world problem with high dimensionality.
• In addition to the weights, the structure of RBMs is also optimized.
• In addition to the load peak, the suggested approach is extended to predict the 24-h load pattern.

3. Determining the Effective Variables

In this section, the effect of each variable on load forecasting is examined. The ANN inputs are altered in different scenarios, and the load peak forecasting error is compared to determine which set of inputs yields the best prediction result. The results of modelling and forecasting are depicted in different diagrams. After identifying the effective inputs, the structure (number of neurons) is also optimized. A summary of the scenarios is given as follows:

1

Load peak, gross domestic product (GDP), temperature peak and humidity percentage peak on a similar day in the past five years; the inflation rate in the past five years; type of day (working day or holiday).

2

Load peak, humidity percentage peak and temperature peak on a similar day in the past five years; type of day.

3

Load peak and the average temperature on a similar day in the past five years; type of day.

4

Load peak and temperature peak on a similar day in the past five years; type of day.

5

Load peak, temperature peak and minimum temperature on a similar day in the past five years; type of day.

6

Peak temperature on a similar day in the past five years; GDP and inflation rate in the past five years; type of day.

7

Average load peak, average peak temperature and peak temperature on a similar day in the past five years; type of day.

8

Load peak and average load peak on a similar month in the past five years.

9

Load peak, average load peak and temperature peak on a similar day in the past five years; average load peak and average temperature peak on a similar month in the past five years; type of day.

10

Load peak on a similar day in the past five years; average load peak on a similar month in the past five years; type of day.

11

Load peak on a similar day in the past five years; average temperature peak on a similar month in the past five years; type of day.

12

Load peak and temperature peak on a similar day in the past five years; average load peak and average temperature peak on a similar month in the past five years; type of day.

13

Load peak and temperature peak on a similar day in the past five years; average load peak, average temperature and average humidity peak on a similar month in the past five years; type of day.

14

Load peak on a similar day in the past five years divided by the average load peak of the same year; temperature peak on a similar day divided by the average temperature peak in the same month in the past five years; type of day.

15

GDP, number of subscribers (NOS), inflation rate and temperature peak in the past five years; type of day.

16

Load peak in the past five years; type of day.

The deep learning algorithm is executed for all scenarios, and the results are given in

Table 1. Results of MAPE for different scenarios—validation data—deep learning algorithm.

Scenario Number	Minimum	Maximum	Average
1	15.4781	23.5555	19.1534
2	16.3671	16.3719	16.3690
3	18.6534	18.6537	18.6536
4	13.1698	13.1699	13.1698
5	17.2136	17.2147	17.2144
6	29.6295	46.0934	39.7988
7	11.6415	11.6415	11.6415
8	13.9718	13.9721	13.9720
9	11.7558	24.0080	15.6033
10	15.9929	15.9930	15.9929
11	12.9877	12.9884	12.9881
12	13.0816	15.0096	14.2075
13	13.0105	15.2048	14.4485
14	11.9303	11.9369	11.9326
15	12.1229	12.9415	12.6579
16	18.2222	18.2222	18.2222

and

Table 2. Results of MAPE for different scenarios—training data—deep learning algorithm.

Scenario Number	Minimum	Maximum	Average
1	11.8630	12.2117	11.9361
2	14.1981	14.2084	14.2022
3	14.3137	14.3140	14.3139
4	11.5422	11.5423	11.5423
5	14.0769	14.0787	14.0782
6	13.6844	13.6859	13.6852
7	8.3699	8.3699	8.3699
8	7.6704	7.6705	7.6704
9	7.3656	8.7696	7.9046
10	7.4343	7.4345	7.4344
11	12.0199	12.0203	12.0201
12	7.4794	8.3951	7.9008
13	7.4701	8.2060	7.7275
14	9.3015	9.3051	9.3038
15	11.4182	11.9852	11.7858
16	14.5071	14.5071	14.5071

. The desired results are acquired for a state where only climatic variables and historical load data are used. The MAPE diagram for test and training data sorted by the best average are illustrated in Figure 1 and Figure 2, respectively. For the entire validation data (one year and 7 months), the minimum MAPE is about 10%.

4. Suggested RBM and Learning Machine

The proposed structure is displayed in Figure 3. The proposed algorithm for training is the contrastive divergence (CD) algorithm. The input variables are explained in the next section. The output of RBM represents the predicted load peak for the specified date. The suggested approach is also extended for 24-h pattern forecasting. The details of computing RBM and the learning method are given as follows:

(1) The degree of probability for apparent and hidden nodes is considered as follows.

$$p\left(v_k,h_k,\xi ,\epsilon ,\delta \right)=\frac{e^{E\left(v_k,h_k,\xi ,\epsilon ,\delta \right)}}{{\displaystyle \sum _{k=1}^n}E\left(v_k,h_k,\xi ,\epsilon ,\delta \right)},$$

(1)

where $\xi ,\epsilon ,\delta$ are trainable parameters, n is the number of training data segments, $v_k$ and $h_k$ are the input vectors in visible and hidden layers, respectively, and E is the cost function.

(2) The energy function is defined as follows [32]:

$$E\left(v_k,h_k,\xi ,\epsilon ,\delta \right)=-h_k^T\xi v_k-{\epsilon }^T{h}_k-{\delta }^T{v}_k.$$

(2)

(3) Initialization:

$$iter=0,\Delta \xi ={\underset{-}{0}}_{\left(n_v\times n_h\right)},\Delta \epsilon ={\underset{-}{0}}_{\left(n_h\times 1\right)},\Delta \delta ={\underset{-}{0}}_{\left(n_v\times 1\right)}.$$

(3)

(4) For input data, $v_0$, values of $[h_0,v_1,h_1]$ are obtained as follows:

$$\begin{array}{c}{h}_0=\sigma \left({\xi }^T{v}_0+\epsilon \right)\hfill \\ v_1=\sigma \left(\xi h_0+\delta \right)\hfill \\ h_1=\sigma \left({\xi }^T{v}_1+\epsilon \right)\hfill \end{array}$$

(4)

$$\sigma \left(x\right)=1/\left(1+exp\left(-x\right)\right).$$

(5)

(5) Parameter changes are obtained using the following equations.

$$\Delta \xi =\gamma \Delta \xi +\frac{\eta }{n}\left(h_0^T{v}_0-h_1^T{v}_1\right)$$

(6)

$$\Delta \epsilon =\gamma \Delta \epsilon +\frac{\eta }{n}{\left(h_0^{}-h_1^{}\right)}^T{\underset{-}{1}}_{\left(n\times 1\right)}$$

(7)

$$\Delta \delta =\gamma \Delta \delta +\frac{\eta }{n}{\left(v_0^{}-v_1^{}\right)}^T{\underset{-}{1}}_{\left(n\times 1\right)}$$

(8)

where $\gamma$ and $\eta$ are constant learning rates.

(6) Update the parameters using the following equations.

$$\xi \left(t+1\right)=\xi \left(t\right)+\Delta \xi -\varsigma \xi \left(t\right)$$

(9)

$$\epsilon \left(t+1\right)=\epsilon \left(t\right)+\Delta \epsilon$$

(10)

$$\delta \left(t+1\right)=\delta \left(t\right)+\Delta \delta$$

(11)

5. Examining the Hourly, Weekdays and Weekends Forecasts

A local neural network is considered for each hour to enhance learning. The suggested structure is shown in Figure 4. The designed scheme leads to a better learning rate, while the effect of climatic information on some hours that are less temperature-dependent is decreased.

The proposed algorithm is generalized to hourly forecasting based on deep learning, similar to the previous section. Similar to Figure 3, a local RBM network is considered for each hour in the deep learning model. Load pattern forecasting for some weeks of April and June and their daily error percentage charts are shown in Figure 5 and Figure 6, respectively.

Similar to weekdays, the proposed algorithm based on deep learning is generalized to the holidays. In the suggested approach, the type of day is not considered as an input variable, but its effect is indirectly considered. In other words, the historical load data as the input variables are considered in similar days of past years. For evaluation, the data from March 2019 to March 2020 are used as the validation data and from March 2010 to March 2018 are used as the training data. Hourly forecast diagram for normal days in the first week of April 2019 and associated MAPE results are given in Figure 7. The hourly forecasts for normal days in spring 2019 and the related MAPE results are given in Figure 8. The hourly forecast diagram for holidays over a year, from March 2019 to March 2020, is shown in Figure 9. The load peak forecast chart for normal days during a year from March 2019 to March 2020 and the related MAPEs are illustrated in Figure 10. As the statistical analysis shows, for most days, the error rate is less than 5%.

6. Suggested Application

By various examinations, the most effective variables are considered to be the weather conditions, type of day and historical load data. A practical application is designed as shown in Figure 11. In this application, the type of inputs, the length of input data, the neurons and the layer number are optimized such that the most accurate performance is achieved. The main aspects of the designed scheme are summarized as:

• Just the weather data, historical load data and calendar (a calendar which shows the type of days in the sense of working day or holiday) are considered as input data.
• To consider the effect of other factors, such as economic factors and population, the case study region is classified into sub-regions. In addition, by considering the pattern of load consumption, the effects of some unavailable and uncertain factors are indirectly considered.
• A simple algorithm is considered to find and correct the bad data. The data of each day are compared with the mean of similar days (similar days are defined as the days that are similar in the sense of working days or holidays, and they are not as far as one month). If the data of one day are far from the average, they are detected as a candidate for bad data. If the weather conditions of this candidate are closer to the mean of similar days, then they are considered bad data.
• In addition to load peak forecasting, the 24-h pattern is also forecast for one year ahead.
• To achieve the most accurate results, in addition to the parameters, the number of neurons, the number of layers, and the type of input variables are also optimized. It should be noted that, in the mid-term forecasting, various economic, social, and cultural factors are effective. Most of these factors are unavailable or uncertain. To consider the effect of these factors in an indirect scheme, the load data in the past dates are considered as input variables, and the length of historical data is optimized.
• For a long period of prediction, to improve the accuracy, it is suggested that for each month, a different structure is optimized. The learning data is divided into some short periods, and for each period a different structure is optimized. For example, suppose that the period of prediction is from January to July one year ahead. The learning data are divided into seven parts, and for each month a different RBM is optimized.

7. Comparisons

In this section, some comparisons are given to better show the suggested approach’s superiority. In [13,16], the ANNs are applied to MTLF. The load peak forecast accuracy is examined for normal days during a year from March 2019 to March 2020. The values of MAPE are given in

Table 3. Comparison of MAPE for different ANN-based methods.

Method	Minimum	Maximum	Average
Method of  [13]	13.474	15.014	14.244
Method of  [16]	11.014	12.841	11.927
Suggested method	3.488	3.524	3.506

. The value of MAPE for the suggested method is remarkably better than conventional ANN-based approaches. In the proposed approach, the set of input variables, the number of layers in RBMs and the number of neurons are optimized to achieve the best accuracy. In addition, the parameters of hidden layers are well tuned by the suggested deep learning approach.

8. Conclusions

In this paper, an experimental approach based on deep learning techniques was designed for load forecasting. The proposed approach was applied to forecast the electrical load for the year ahead. In addition to optimizing the free parameters of RBMs, the number of neurons and layers and the input set’s optimality were investigated by a statistical analysis approach. To determine the most effective input set of RBM, the impact of different variables, such as temperature, historical load data, type of days, date and economic factors were analysed under various conditions. In addition, the 24-h pattern was also predicted to forecast the load peak. The suggested approach can predict both working days and holidays. Through various simulations, the forecasting accuracy of the designed approach was examined by real-world load data, and the accuracy for one year ahead was estimated to be more than 95%. The designed software applies various optimizations to the set of input variables and the structure of RBMs to obtain the most accurate performance. However, the main limitation is that there is no analysis of the datasets, and wrong data are not detected. For future studies, besides the pre-analysis of datasets to remove the wrong data, the designed application can also be developed for long-term forecasting.