# Deep-Reinforcement-Learning-Based Low-Carbon Economic Dispatch for Community-Integrated Energy System under Multiple Uncertainties

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Plant Model

#### 2.1. Environmental Model

#### 2.1.1. Electricity–Gas Coupling Model on the Energy Supply Side

- (1)
- Gas turbine model

- (2)
- P2G equipment model

#### 2.1.2. Electric–Cooling Coupling Model on the Energy Supply Side

#### 2.1.3. Energy Storage Device Model on the Energy Storage Side

#### 2.1.4. Demand Response Model Based on User’s Willingness on the Energy Demand Side

- (1)
- Curtailable electric/gas load model in DRUW

- (2)
- shiftable electric/gas load model in DRUW

#### 2.1.5. Community Electric Vehicle Model on the Energy Demand Side

#### 2.2. Agent Model

#### 2.2.1. Markov Decision Process for CIES Dispatch

_{1}in state s

_{1}and the state transforming to s

_{2}; R means the reward given by the environment after the agent makes the action; and γ means the discount factor, which means the degree of influence of the reward obtained in future periods on the cumulative reward.

#### 2.2.2. SAC Deep Reinforcement Learning Algorithm

#### 2.2.3. Agent Observation Space

#### 2.2.4. Agent Action Space

#### 2.2.5. Agent Reward Function

- (1)
- CIES revenue from electricity sales

- (2)
- CIES revenue from gas sales

- (3)
- Cost of gas turbine pollution emissions

- (4)
- Cost of Carbon trading

_{2}absorption coefficient of the P2G equipment; ${e}^{\mathrm{GT}}$ and ${e}^{\mathrm{grid}}$ are the unit carbon emission allowances for the gas turbine and grid, respectively.

- (5)
- Cost of purchasing electricity/gas

- (6)
- Cost of dispatch curtailable electric/gas load

- (7)
- Cost of dispatch shiftable electric/gas load

## 3. Model Training

#### 3.1. Construction of the Training Scenario Set

#### 3.2. Construction of the Training Scenario Set

## 4. Case Studies

#### 4.1. Experimental Setup

#### 4.2. Simulation Results

#### 4.3. Generalization Performance Analysis under Source and Load Uncertain Scenarios

#### 4.4. Generalization Performance Analysis under Outdoor Temperature Uncertain Scenarios

#### 4.5. Generalization Performance Analysis under Uncertain Scenarios of Electric Vehicle Trips

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Wang, C.S.; Lv, C.X.; Li, P.; Song, G.Y.; Li, S.Q.; Xu, X.D.; Wu, J.Z. Modeling and optimal operation of community integrated energy systems: A case study from China. Appl. Energy
**2018**, 230, 1242–1254. [Google Scholar] [CrossRef] - Zhou, Y.Z.; Wei, Z.N.; Sun, G.Q.; Cheung, K.W.; Zang, H.X.; Chen, S. A robust optimization approach for integrated community energy system in energy and ancillary service markets. Energy
**2018**, 148, 1–15. [Google Scholar] [CrossRef] - Li, Y.; Wang, B.; Yang, Z.; Li, J.Z.; Chen, C. Hierarchical stochastic scheduling of multi-community integrated energy systems in uncertain environments via Stackelberg game. Appl. Energy
**2022**, 308, 118392. [Google Scholar] [CrossRef] - Zhang, Y.Y.; Zhao, H.R.; Li, B.K.; Wang, X.J. Research on dynamic pricing and operation optimization strategy of integrated energy system based on Stackelberg game. Int. J. Electr. Power Energy Syst.
**2022**, 143, 108446. [Google Scholar] [CrossRef] - Yang, S.B.; Tan, Z.F.; Zhou, J.H.; Xue, F.; Gao, H.D.; Lin, H.Y.; Zhou, F.A. A two-level game optimal dispatching model for the park integrated energy system considering Stackelberg and cooperative games. Int. J. Electr. Power Energy Syst.
**2021**, 130, 106959. [Google Scholar] [CrossRef] - Gao, H.; Li, Z.S. A Benders Decomposition Based Algorithm for Steady-State Dispatch Problem in an Integrated Electricity-Gas System. IEEE Trans. Power Syst.
**2021**, 36, 3817–3820. [Google Scholar] [CrossRef] - Luo, X.J.; Fong, K.F. Development of integrated demand and supply side management strategy of multi-energy system for residential building application. Appl. Energy
**2019**, 242, 570–587. [Google Scholar] [CrossRef] - Li, P.; Wang, Z.X.; Wang, N.; Yang, W.H.; Li, M.Z.; Zhou, X.C.; Yin, Y.X.; Wang, J.H.; Guo, T.Y. Stochastic robust optimal operation of community integrated energy system based on integrated demand response. Int. J. Electr. Power Energy Syst.
**2021**, 128, 106735. [Google Scholar] [CrossRef] - Liu, G.; Qin, Z.F.; Diao, T.Y.; Wang, X.W.; Wang, P.M.; Bai, X.Q. Low carbon economic dispatch of biogas-wind-solar renewable energy system based on robust stochastic optimization. Int. J. Electr. Power Energy Syst.
**2022**, 139, 108069. [Google Scholar] [CrossRef] - Yan, R.J.; Wang, J.J.; Wang, J.H.; Tian, L.; Tang, S.Q.; Wang, Y.W.; Zhang, J.; Cheng, Y.L.; Li, Y. A two-stage stochastic-robust optimization for a hybrid renewable energy CCHP system considering multiple scenario-interval uncertainties. Energy
**2022**, 247, 123498. [Google Scholar] [CrossRef] - Li, X.Q.; Zhang, L.Z.; Wang, R.Q.; Sun, B.; Xie, W.J. Two-Stage Robust Optimization Model for Capacity Configuration of Biogas-Solar-Wind Integrated Energy System. IEEE Trans. Ind. Appl.
**2023**, 59, 662–675. [Google Scholar] [CrossRef] - Shuvo, S.S.; Yilmaz, Y. Home Energy Recommendation System (HERS): A Deep Reinforcement Learning Method Based on Residents' Feedback and Activity. IEEE Trans. Smart Grid
**2022**, 13, 2812–2821. [Google Scholar] [CrossRef] - Lu, R.Z.; Bai, R.C.; Luo, Z.; Jiang, J.H.; Sun, M.Y.; Zhang, H.T. Deep reinforcement learning-based demand response for smart facilities energy management. IEEE Trans. Ind. Electron.
**2021**, 69, 8554–8565. [Google Scholar] [CrossRef] - Ren, M.F.; Liu, X.F.; Yang, Z.L.; Zhang, J.H.; Guo, Y.J.; Jia, Y.B. A novel forecasting based scheduling method for household energy management system based on deep reinforcement learning. Sustain. Cities Soc.
**2022**, 76, 103207. [Google Scholar] [CrossRef] - Lai, B.C.; Chiu, W.Y.; Tsai, Y.P. Multiagent Reinforcement Learning for Community Energy Management to Mitigate Peak Rebounds Under Renewable Energy Uncertainty. IEEE Trans. Emerg. Top. Comput. Intell.
**2022**, 6, 568–579. [Google Scholar] [CrossRef] - Ye, Y.J.; Qiu, D.W.; Wu, X.D.; Strbac, G.; Ward, J. Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning. IEEE Trans. Smart Grid.
**2020**, 11, 3068–3082. [Google Scholar] [CrossRef] - Ding, H.Y.; Xu, Y.; Hao, B.C.S.; Li, Q.Q.; Lentzakis, A. A safe reinforcement learning approach for multi-energy management of smart home. Electr. Power Syst. Res.
**2022**, 210, 108120. [Google Scholar] [CrossRef] - Xue, X.; Wang, J.X.; Zhang, Y.; Yong, W.Z.; Qi, J.; Li, H.T. Model-data-event based community integrated energy system low-carbon economic scheduling. Renew. Sustain. Energy Rev.
**2023**, 182, 113379. [Google Scholar] [CrossRef] - Qiu, Y.; Zhou, S.Y.; Xia, D.; Gu, W.; Sun, K.Y.; Han, G.Y.; Zhang, K.; Lv, H.K. Local integrated energy system operational optimization considering multi-type uncertainties: A reinforcement learning approach based on improved TD3 algorithm. IET Renew. Power Gener.
**2023**, 17, 2236–2256. [Google Scholar] [CrossRef] - Hong, S.H.; Lee, H.S. Robust Energy Management System with Safe Reinforcement Learning Using Short-Horizon Forecasts. IEEE Trans. Smart Grid.
**2023**, 14, 2485–2488. [Google Scholar] [CrossRef] - Liu, Y.; Liu, T.Y. Research on System Planning of Gas-Power Integrated System Based on Improved Two-Stage Robust Optimization and Non-Cooperative Game Method. IEEE Access
**2021**, 9, 79169–79181. [Google Scholar] [CrossRef] - Li, G.Q.; Zhang, R.F.; Jiang, T.; Chen, H.H.; Bai, L.Q.; Li, X.J. Security-constrained bi-level economic dispatch model for integrated natural gas and electricity systems considering wind power and power-to-gas process. Appl. Energy
**2017**, 194, 696–704. [Google Scholar] [CrossRef] - Sun, G.Q.; Qian, W.H.; Huang, W.J.; Xu, Z.; Fu, Z.X.; Wei, Z.N.; Chen, S. Stochastic Adaptive Robust Dispatch for Virtual Power Plants Using the Binding Scenario Identification Approach. Energies
**2019**, 12, 1918. [Google Scholar] [CrossRef] - Li, Y.; Zou, Y.; Tan, Y.; Cao, Y.J.; Liu, X.D.; Shahidehpour, M.; Tian, S.M.; Bu, F.P. Optimal Stochastic Operation of Integrated Low-Carbon Electric Power, Natural Gas, and Heat Delivery System. IEEE Trans. Sustain. Energy
**2018**, 9, 273–283. [Google Scholar] [CrossRef] - Zhang, B.; Hu, W.H.; Cao, D.; Huang, Q.; Chen, Z.; Blaabjerg, F. Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energy Convers. Manag.
**2019**, 202, 112199. [Google Scholar] [CrossRef] - Dong, J.; Wang, H.X.; Yang, J.Y.; Lu, X.Y.; Gao, L.; Zhou, X.R. Optimal Scheduling Framework of Electricity-Gas-Heat Integrated Energy System Based on Asynchronous Advantage Actor-Critic Algorithm. IEEE Access
**2021**, 9, 139685–139696. [Google Scholar] [CrossRef] - Zhang, B.; Hu, W.H.; Cao, D.; Huang, Q.; Chen, Z.; Blaabjerg, F. Economical operation strategy of an integrated energy system with wind power and power to gas technology—A DRL-based approach. IET Renew. Power Gener.
**2020**, 14, 3292–3299. [Google Scholar] [CrossRef] - Boutilier, C.; Dean, T.; Hanks, S. Decision-theoretic planning: Structural assumptions and computational leverage. J. Artif. Intell. Res.
**1999**, 11, 1–94. [Google Scholar] [CrossRef] - Kober, J.; Bagnell, J.A.; Peters, J. Reinforcement learning in robotics: A survey. Int. J. Rob. Res.
**2013**, 32, 1238–1274. [Google Scholar] [CrossRef] - Han, X.Y.; Mu, C.X.; Yan, J.; Niu, Z.Y. An autonomous control technology based on deep reinforcement learning for optimal active power dispatch. Int. J. Electr. Power Energy Syst.
**2023**, 145, 108686. [Google Scholar] [CrossRef] - Xiao, B.Y.; Yang, W.W.; Wu, J.M.; Walker, P.D.; Zhang, N. Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle. Energy
**2022**, 253, 124105. [Google Scholar] [CrossRef] - Yang, X.H.; Zhang, Z.L.; Mei, L.H.; Wang, X.P.; Deng, Y.H.; Wei, S.; Liu, X.P. Optimal configuration of improved integrated energy system based on stepped carbon penalty response and improved power to gas. Energy
**2023**, 263, 125985. [Google Scholar] [CrossRef] - Mei, F.; Zhang, J.T.; Lu, J.X.; Lu, J.J.; Jiang, Y.H.; Gu, J.Q.; Yu, K.; Gan, L. Stochastic optimal operation model for a distributed integrated energy system based on multiple-scenario simulations. Energy
**2021**, 219, 119629. [Google Scholar] [CrossRef] - Wang, J.J.; Huo, S.J.; Yan, R.J.; Cui, Z.H. Leveraging heat accumulation of district heating network to improve performances of integrated energy system under source-load uncertainties. Energy
**2022**, 252, 124002. [Google Scholar] [CrossRef] - Wais, P. A review of Weibull functions in wind sector. Renew. Sustain. Energy Rev.
**2017**, 70, 1099–1107. [Google Scholar] [CrossRef] - Ettoumi, F.Y.; Mefti, A.; Adane, A.; Bouroubi, M.Y. Statistical analysis of solar measurements in Algeria using beta distributions. Renew. Energy
**2002**, 26, 47–67. [Google Scholar] [CrossRef] - Das, S.; Malakar, T. Estimating the impact of uncertainty on optimum capacitor placement in wind-integrated radial distribution system. Int. Trans. Electr. Energy Syst.
**2020**, 30, e12451. [Google Scholar] [CrossRef] - Liu, D.Q. Cluster Control for EVs Participating in Grid Frequency Regulation by Using Virtual Synchronous Machine with Optimized Parameters. Appl. Sci.
**2019**, 9, 1924. [Google Scholar] [CrossRef]

**Figure 1.**Model framework of community-integrated energy system based on deep reinforcement learning.

**Figure 4.**Training curve under different temperature parameters: (

**a**) return value; (

**b**) carbon trading costs.

**Figure 9.**Experimental data: (

**a**) uncertain scenario set of outdoor temperature; (

**b**) dispatch results of indoor temperature.

Parameter | Value | Parameter | Value | Parameter | Value |
---|---|---|---|---|---|

${p}_{\mathrm{max}}^{\mathrm{GT}}$(kW) | 100 | ${E}_{\mathrm{max}}^{\mathrm{ES},\mathrm{AC}}$(kWh) | 50 | ${\eta}^{\mathrm{ESch},\mathrm{BA}}$ | 95% |

${p}_{\mathrm{min}}^{\mathrm{GT}}$(kW) | 10 | ${E}_{\mathrm{min}}^{\mathrm{ES},\mathrm{AC}}$(kWh) | 0 | ${\eta}^{\mathrm{ESdis},\mathrm{BA}}$ | 95% |

$\Delta {P}^{\mathrm{GTmax}}$(kW/h) | 70 | ${E}_{\mathrm{max}}^{\mathrm{ES},\mathrm{EV}}$(kWh) | 20 | ${\eta}^{\mathrm{ESch},\mathrm{GS}}$ | 95% |

${m}^{{\mathrm{GTSO}}_{\mathrm{X}}}$ | 0.0098 | ${E}_{\mathrm{min}}^{\mathrm{ES},\mathrm{EV}}$(kWh) | 6 | ${\eta}^{\mathrm{ESdis},\mathrm{GS}}$ | 95% |

${m}^{{\mathrm{GTNO}}_{\mathrm{X}}}$ | 0.543 | ${P}_{\mathrm{max}}^{\mathrm{ESch},\mathrm{BA}}$(kW) | 30 | ${\eta}^{\mathrm{ESch},\mathrm{EV}}$ | 95% |

${a}_{\mathrm{g}}$ | 0.11 | ${P}_{\mathrm{max}}^{\mathrm{ESdis},\mathrm{BA}}$(kW) | 30 | ${\eta}^{\mathrm{ESdis},\mathrm{EV}}$ | 95% |

${b}_{\mathrm{g}}$ | 2 | ${P}_{\mathrm{max}}^{\mathrm{ESch},\mathrm{GS}}$(m^{3}) | 50 | ${\mu}^{\mathrm{ACc}}$ | 2.6 |

${c}_{\mathrm{g}}$ | 0 | ${P}_{\mathrm{max}}^{\mathrm{ESdis},\mathrm{GS}}$(m^{3}) | 50 | ${\mu}^{\mathrm{ACs}}$ | 0.0045 |

${E}_{\mathrm{max}}^{\mathrm{ES},\mathrm{BA}}$(kWh) | 100 | ${P}_{\mathrm{max}}^{\mathrm{ESch},\mathrm{AC}}$(kW) | 20 | ${\mu}^{\mathrm{ACr}}$ | 0.0038 |

${E}_{\mathrm{min}}^{\mathrm{ES},\mathrm{BA}}$(kWh) | 10 | ${P}_{\mathrm{max}}^{\mathrm{ESdis},\mathrm{AC}}$(kW) | 20 | ${H}_{\mathrm{max}}^{\mathrm{ACc}}$ | 25 |

${E}_{\mathrm{max}}^{\mathrm{ES},\mathrm{GS}}$(m^{3}) | 150 | ${P}_{\mathrm{max}}^{\mathrm{ESch},\mathrm{EV}}$(kW) | 8 | ${\zeta}^{\mathrm{EV}}$ | 0.241 |

${E}_{\mathrm{min}}^{\mathrm{ES},\mathrm{GS}}$(m^{3}) | 10 | ${P}_{\mathrm{max}}^{\mathrm{ESdis},\mathrm{EV}}$(kW) | 8 |

Time Period | Electricity Price (USD/kWh) | Natural Gas Prices (USD/m^{3}) |
---|---|---|

Peak section | 0.143 | 0.043 |

Flat section | 0.114 | 0.036 |

Valley section | 0.086 | 0.029 |

Time Steps per Episode | Learning Rate | Discount Factor | Batch Size | Replay Buffer Size | Soft Update Factor |
---|---|---|---|---|---|

24 | 0.0003 | 0.998 | 256 | 1,000,000 | 0.005 |

Case Index | Realization DRUW | Response Damping Coefficient | Demand Response Uncertainty Interval |
---|---|---|---|

1 | × | — | — |

2 | √ | 1.0 | [0.5,0.6] |

3 | √ | 1.3 | [0.5,0.6] |

4 | √ | 1.0 | [0.5–0.7] |

Case Index | Electric Load Curtailment (kWh) | Gas Load Curtailment (m^{3}) | Demand Response Compensation Costs (USD) | Cost of Electricity/Gas Purchase (USD) | CIES Net Revenue (USD) |
---|---|---|---|---|---|

1 | 0 | 0 | 0 | 183.34 | 1287.00 |

2 | 101.00 | 39.66 | 7.87 | 170.83 | 1368.32 |

3 | 2.29 | 3.35 | 2.13 | 187.55 | 1303.95 |

4 | 28.18 | 1.34 | 3.17 | 180.49 | 1316.06 |

**Table 6.**Economic indicators of CIES dispatch results under the source and load uncertain scenarios.

Source and Load Fluctuation Rate | Cost of Electricity Purchase (USD) | Cost of Gas Purchase (USD) | Carbon Trading Costs (USD) | Demand Response Compensation Costs (USD) | CIES Net Revenue (USD) |
---|---|---|---|---|---|

5% | 65.92 | 101.41 | 3.41 | 8.06 | 1403.97 |

10% | 54.89 | 101.14 | 2.82 | 7.69 | 1392.19 |

15% | 62.76 | 102.08 | 3.37 | 7.76 | 1392.87 |

Scenario Index | Number of Trips | Trip Time | Trip Mileage |
---|---|---|---|

1 | 1 | 7:00–16:00 | 24 |

2 | 1 | 9:00–18:00 | 20 |

3 | 2 | 7:00–13:00 16:00–18:00 | 12 16 |

Scenario Index | Net Charging Volume (kWh) | Surplus Power Storage (kWh) | CIES Net Revenue (USD) |
---|---|---|---|

1 | 6.93 | 14.22 | 1368.92 |

2 | 5.00 | 15.18 | 1360.35 |

3 | 6.84 | 16.15 | 1355.24 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Mo, M.; Xiong, X.; Wu, Y.; Yu, Z.
Deep-Reinforcement-Learning-Based Low-Carbon Economic Dispatch for Community-Integrated Energy System under Multiple Uncertainties. *Energies* **2023**, *16*, 7669.
https://doi.org/10.3390/en16227669

**AMA Style**

Mo M, Xiong X, Wu Y, Yu Z.
Deep-Reinforcement-Learning-Based Low-Carbon Economic Dispatch for Community-Integrated Energy System under Multiple Uncertainties. *Energies*. 2023; 16(22):7669.
https://doi.org/10.3390/en16227669

**Chicago/Turabian Style**

Mo, Mingshan, Xinrui Xiong, Yunlong Wu, and Zuyao Yu.
2023. "Deep-Reinforcement-Learning-Based Low-Carbon Economic Dispatch for Community-Integrated Energy System under Multiple Uncertainties" *Energies* 16, no. 22: 7669.
https://doi.org/10.3390/en16227669