# Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

- 1.
- Syncing the exogenous information on weather data with the electricity market information, i.e., prices and demand data, to create an extensive dataset. The significance of leveraging the external dataset is illustrated, and the importance of features is also demonstrated.
- 2.
- Both the DAM and RTM are analyzed for price prediction. A realistic set of assumptions is made regarding the availability of features for both the RTM and DAM once the prices are predicted 24–36 h in advance for the following market operation day upon the clearing of the market.
- 3.
- The ensemble learning method, namely the Random Forest (RF), is used to calculate the probability distribution of the predicted market prices for the DAM and RTM, as well as the gap.
- 4.
- An LSTM architecture is deployed to enhance predictions given the complexity of predicting values for the time series dataset. The proposed model is compared with other statistical machine learning methods, which demonstrate significant improvements.

## 2. Learning Algorithms and Methodologies

#### 2.1. Least Absolute Shrinkage and Selection Operator (LASSO)

#### 2.2. Support Vector Regression (SVR)

#### 2.3. Random Forest Algorithm

#### 2.4. Long Short-Term Memory (LSTM) Networks

## 3. Prediction Performance Evaluation

## 4. Data Preparation

#### 4.1. Data Collection

#### 4.2. Data Cleansing and Pre-Processing

## 5. Simulation Results

#### 5.1. Feature Importance

#### 5.2. Hyper-Parameter Tuning

#### 5.3. Analysis of Probability Distributions

#### 5.4. Performance Evaluation

#### 5.5. Importance of Exogenous Weather Information

## 6. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Stoft, S. Power System Economics: Designing Markets for Electricity; IEEE Press: Piscataway, NJ, USA, 2002; Volume 468. [Google Scholar]
- Woo, C.K.; Moore, J.; Schneiderman, B.; Ho, T.; Olson, A.; Alagappan, L.; Chawla, K.; Toyama, N.; Zarnikau, J. Merit-order effects of renewable energy and price divergence in California’s day-ahead and real-time electricity markets. Energy Policy
**2016**, 92, 299–312. [Google Scholar] [CrossRef] - Khoshjahan, M.; Baembitov, R.; Kezunovic, M. Impacts of weather-related outages on DER participation in the wholesale market energy and ancillary services. In Proceedings of the CIGRE Grid of the Future Symposium, Providence, RI, USA, 17–20 October 2021. [Google Scholar]
- Li, Y.; Flynn, P. Electricity deregulation, spot price patterns and demand-side management. Energy
**2006**, 31, 908–922. [Google Scholar] [CrossRef] - Benini, M.; Marracci, M.; Pelacchi, P.; Venturini, A. Day-ahead market price volatility analysis in deregulated electricity markets. In Proceedings of the IEEE Power Engineering Society Summer Meeting, Chicago, IL, USA, 21–25 July 2002; Volume 3, pp. 1354–1359. [Google Scholar]
- Veit, D.J.; Weidlich, A.; Yao, J.; Oren, S.S. Simulating the dynamics in two-settlement electricity markets via an agent-based approach. Int. J. Manag. Sci. Eng. Manag.
**2006**, 1, 83–97. [Google Scholar] [CrossRef] - Yao, J.; Adler, I.; Oren, S.S. Modeling and computing two-settlement oligopolistic equilibrium in a congested electricity network. Oper. Res.
**2008**, 56, 34–47. [Google Scholar] [CrossRef] - Bayani, R.; Soofi, A.F.; Manshadi, S.D. Strategic Competition of Electric Vehicle Charging Stations in a Regulated Retail Electricity Market. arXiv
**2021**, arXiv:2111.11592. [Google Scholar] - Mehdipourpicha, H.; Bo, R. Optimal Bidding Strategy for Physical Market Participants With Virtual Bidding Capability in Day-Ahead Electricity Markets. IEEE Access
**2021**, 9, 85392–85402. [Google Scholar] [CrossRef] - Kohansal, M.; Sadeghi-Mobarakeh, A.; Manshadi, S.D.; Mohsenian-Rad, H. Strategic Convergence Bidding in Nodal Electricity Markets: Optimal Bid Selection and Market Implications. IEEE Trans. Power Syst.
**2020**, 36, 891–901. [Google Scholar] [CrossRef] - Kohansal, M.; Samani, E.; Mohsenian-Rad, H. Understanding the Structural Characteristics of Convergence Bidding in Nodal Electricity Markets. IEEE Trans. Ind. Informatics
**2020**, 17, 124–134. [Google Scholar] [CrossRef] - Fan, S.; Liao, J.R.; Kaneko, K.; Chen, L. An integrated machine learning model for day-ahead electricity price forecasting. In Proceedings of the 2006 IEEE PES Power Systems Conference and Exposition, PSCE 2006, Atlanta, GA, USA, 29 October–1 November 2006; pp. 1643–1649. [Google Scholar] [CrossRef]
- Contreras, J.; Espínola, R.; Nogales, F.J.; Conejo, A.J. ARIMA models to predict next-day electricity prices. IEEE Trans. Power Syst.
**2003**, 18, 1014–1020. [Google Scholar] [CrossRef] - Sadeghi-Mobarakeh, A.; Kohansal, M.; Papalexakis, E.E.; Mohsenian-Rad, H. Data mining based on random forest model to predict the California ISO day-ahead market prices. In Proceedings of the 2017 IEEE Power and Energy Society Innovative Smart Grid Technologies Conference, ISGT 2017, Washington, DC, USA, 23–26 April 2017. [Google Scholar] [CrossRef]
- Weron, R. Electricity price forecasting: A review of the state-of-the-art with a look into the future. Int. J. Forecast.
**2014**, 30, 1030–1081. [Google Scholar] [CrossRef] - Gao, G.; Lo, K.; Fan, F. Comparison of ARIMA and ANN models used in electricity price forecasting for power market. Energy Power Eng.
**2017**, 9, 120–126. [Google Scholar] [CrossRef] - Adebiyi, A.A.; Adewumi, A.O.; Ayo, C.K. Comparison of ARIMA and artificial neural networks models for stock price prediction. J. Appl. Math.
**2014**, 2014, 614342. [Google Scholar] [CrossRef] - Bengio, Y.; Courville, A.; Vincent, P. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 1798–1828. [Google Scholar] [CrossRef] - He, K.; Yang, Q.; Ji, L.; Pan, J.; Zou, Y. Financial Time Series Forecasting with the Deep Learning Ensemble Model. Mathematics
**2023**, 11, 1054. [Google Scholar] [CrossRef] - Baldo, A.; Cuzzocrea, A.; Fadda, E.; Bringas, P.G. Financial forecasting via deep-learning and machine-learning tools over two-dimensional objects transformed from time series. In Proceedings of the Hybrid Artificial Intelligent Systems: 16th International Conference, HAIS 2021, Bilbao, Spain, 22–24 September 2021; pp. 550–563. [Google Scholar]
- Zheng, J.; Xu, C.; Zhang, Z.; Li, X. Electric load forecasting in smart grids using Long-Short-Term-Memory based Recurrent Neural Network. In Proceedings of the 2017 51st Annual Conference on Information Sciences and Systems, CISS 2017, Baltimore, MD, USA, 22–24 March 2017; pp. 1–6. [Google Scholar] [CrossRef]
- Bouktif, S.; Fiaz, A.; Ouni, A.; Serhani, M.A. Optimal deep learning LSTM model for electric load forecasting using feature selection and genetic algorithm: Comparison with machine learning approaches. Energies
**2018**, 11, 1636. [Google Scholar] [CrossRef] - Jiang, L.; Hu, G. Day-Ahead Price Forecasting for Electricity Market using Long-Short Term Memory Recurrent Neural Network. In Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision, ICARCV 2018, Singapore, 18–21 November 2018; pp. 949–954. [Google Scholar] [CrossRef]
- Aggarwal, S.K.; Saini, L.M.; Kumar, A. Electricity price forecasting in deregulated markets: A review and evaluation. Int. J. Electr. Power Energy Syst.
**2009**, 31, 13–22. [Google Scholar] [CrossRef] - Ji, Y.; Kim, J.; Thomas, R.J.; Tong, L. Forecasting real-time locational marginal price: A state space approach. In Proceedings of the Conference Record—2013 Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 3–6 November 2013; pp. 379–383. [Google Scholar] [CrossRef]
- Mujeeb, S.; Javaid, N.; Ilahi, M.; Wadud, Z.; Ishmanov, F.; Afzal, M.K. Deep long short-term memory: A new price and load forecasting scheme for big data in smart cities. Sustainability
**2019**, 11, 987. [Google Scholar] [CrossRef] - Zhang, Z.; Wu, M. Predicting real-time locational marginal prices: A GAN-based video prediction approach. arXiv
**2020**, arXiv:2003.09527. [Google Scholar] - Zahid, M.; Ahmed, F.; Javaid, N.; Abbasi, R.A.; Kazmi, H.S.Z.; Javaid, A.; Bilal, M.; Akbar, M.; Ilahi, M. Electricity price and load forecasting using enhanced convolutional neural network and enhanced support vector regression in smart grids. Electronics
**2019**, 8, 122. [Google Scholar] [CrossRef] - Tibshirani, R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Methodol.
**1996**, 58, 267–288. [Google Scholar] [CrossRef] - Breiman, L. Bagging predictors. Mach. Learn.
**1996**, 24, 123–140. [Google Scholar] [CrossRef] - Breiman, L. Random forests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef] - Siegelmann, H.T.; Sontag, E.D. Turing computability with neural nets. Appl. Math. Lett.
**1991**, 4, 77–80. [Google Scholar] [CrossRef] - Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed]

**Figure 2.**The probability distribution of the DAM and RTM price predictions for a specific date and time, procured by the RF algorithm.

**Figure 3.**A comparison between the probability distribution of direct gap predictions and the difference between separately predicted prices for the DAM and RTM at 8 a.m., procured by LSTM network.

**Figure 4.**A comparison between the probability distribution of direct gap predictions and the difference between separately predicted prices for the DAM and RTM at 5 pm, procured by LSTM network.

**Figure 5.**A comparison of predicting the gap using LSTM and RF algorithms for the next 96 h to the actual values of the gap.

**Figure 6.**A comparison of the relative error of direct gap predictions procured by LSTM and RF algorithms for the next 96 h.

MAE | $\left[\frac{1}{s}{\sum}_{t=1}^{s}|{y}_{t}-{\widehat{y}}_{t}|\right]$ |

RMSE | ${\sqrt{\frac{1}{s}{\sum}_{t=1}^{s}{({y}_{t}-{\widehat{y}}_{t})}^{2}}}_{}$ |

nRMSE | ${\sqrt{\frac{1}{s}{\sum}_{t=1}^{s}{({y}_{t}-{\widehat{y}}_{t})}^{2}}}_{}/\left[{y}_{\mathrm{max}}-{y}_{\mathrm{min}}\right]$ |

Max Error | $max\left(\left|{y}_{t}-{\widehat{y}}_{t}\right|\right)\forall t\in \{0,s\}$ |

Feature | Importance Coefficient |
---|---|

GAP LMP price 24 h before | 3.32 |

DAM LMP price 24 h before | 1.9 |

RTM LMP price 24 h before | 1.3 |

Solar Zenith Angle | 1.0 |

Demand Forecast Day-Ahead | 0.78 |

Relative humidity | 0.67 |

Perceptible Water | 0.61 |

Cloud Layer | 0.54 |

Dew point | 0.53 |

Wind Speed | 0.47 |

Wind Direction | 0.46 |

Demand Forecast 2 Days Ahead | 0.46 |

DHI | 0.39 |

Statistics | Gap | DAM | RTM |
---|---|---|---|

Mean | 3.1 | 39.1 | 36.0 |

Standard Deviation | 87.7 | 45.6 | 84.5 |

Min | −1488.6 | −61.6 | −262.3 |

Max | 2276.2 | 2374.4 | 1545.4 |

25th Percentile | −0.16 | 25.0 | 19.6 |

50th Percentile | 6.6 | 33.4 | 27.1 |

75th Percentile | 17.5 | 46.5 | 36.7 |

Error Measure | LASSO | SVR | RF | LSTM |
---|---|---|---|---|

MAE | 9.6 | 11.7 | 5.1 | 4.9 |

RMSE | 13.3 | 33.7 | 7.9 | 7.1 |

nRMSE [%] | 7.4 | 39.8 | 4.4 | 4.2 |

Max Error | 95.2 | 122.7 | 62 | 40 |

Error Measure | LASSO | SVR | RF | LSTM |
---|---|---|---|---|

MAE | 18.9 | 21.4 | 26.4 | 21.2 |

RMSE | 59 | 54 | 71.9 | 48 |

nRMSE [%] | 5.1 | 5.0 | 6.2 | 4.4 |

Max Error | 1064 | 1060 | 1058 | 1040 |

Error Measure | LASSO | SVR | RF | LSTM |
---|---|---|---|---|

MAE | 19.6 | 28.2 | 24.5 | 17.1 |

RMSE | 58.9 | 80.4 | 67.5 | 56.9 |

nRMSE [%] | 4.98 | 6.1 | 5.7 | 4.8 |

Max Error | 1054.8 | 1051 | 1048 | 1046 |

Error Measure | LASSO | SVR | RF | LSTM |
---|---|---|---|---|

MAE | 19.7 | 44.8 | 24.7 | 31.8 |

RMSE | 64 | 81.6 | 75 | 62.15 |

nRMSE [%] | 5.4 | 6.9 | 6.3 | 5.2 |

Max Error | 1053.7 | 1069 | 1058 | 1052 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Nizharadze, N.; Farokhi Soofi, A.; Manshadi, S.
Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data. *Algorithms* **2023**, *16*, 508.
https://doi.org/10.3390/a16110508

**AMA Style**

Nizharadze N, Farokhi Soofi A, Manshadi S.
Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data. *Algorithms*. 2023; 16(11):508.
https://doi.org/10.3390/a16110508

**Chicago/Turabian Style**

Nizharadze, Nika, Arash Farokhi Soofi, and Saeed Manshadi.
2023. "Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data" *Algorithms* 16, no. 11: 508.
https://doi.org/10.3390/a16110508