Environment-Friendly Power Scheduling Based on Deep Contextual Reinforcement Learning

Abstract

A novel approach to power scheduling is introduced, focusing on minimizing both economic and environmental impacts. This method utilizes deep contextual reinforcement learning (RL) within an agent-based simulation environment. Each generating unit is treated as an independent, heterogeneous agent, and the scheduling dynamics are formulated as Markov decision processes (MDPs). The MDPs are then used to train a deep RL model to determine optimal power schedules. The performance of this approach is evaluated across various power systems, including both small-scale and large-scale systems with up to 100 units. The results demonstrate that the proposed method exhibits superior performance and scalability in handling power systems with a larger number of units.

1. Introduction

Electrical energy generated from fossil fuels emits significant amounts of greenhouse gases (GHGs), including carbon dioxide (CO₂), sulfur dioxide (SO₂), and nitrous oxide (N₂O). These emissions have detrimental effects on human health and contribute to climate change and global warming. However, the prevailing focus on economic concerns in power generation often results in higher emission levels since economic costs and environmental impacts tend to be inversely related. This effect has been amplified since the implementation of the global emissions trading scheme (ETS) in 2005, aimed at controlling GHGs [1,2]. As a result, power generation based solely on economic costs leads to increased financial penalties for emissions, along with adverse environmental impacts. Therefore, the traditional approach of scheduling power generation based solely on economic costs, which overlooks environmental ETS, is no longer acceptable [2]. Consequently, it becomes imperative to determine an efficient and environmentally friendly power generation schedule that might have lower emission costs, indicating that it generates fewer GHGs or other pollutants during its operation. The primary objective of environmentally friendly power scheduling is to achieve a sustainable cost and emission balance, where economic concerns are considered without compromising environmental sustainability. This means effectively meeting electricity demand while minimizing the environmental impact, especially in terms of greenhouse gas emissions (GHGs) and other pollutants. By adopting this approach, not only does the profitability of power generation increase, but it also leads to reduced emission levels through efficient management and scheduling of generating units [1,2].

Prior research has explored various model-based approaches, such as conventional and dynamic programming and stochastic optimizations, to address the challenges in power scheduling [3]. However, these methods often face the curse of the dimensionality problem, which leads to the use of heuristic rules and simplifications that may not effectively handle real-sized problems [3,4,5]. As power systems continue to grow in size and complexity, even small improvements in efficiency achieved through enhanced power scheduling methods can yield significant economic and environmental benefits [4].

Recently, artificial intelligence (AI) has shown promise in learning optimal strategies without prior knowledge. Particularly, reinforcement learning (RL) can achieve this by employing self-play learning and adapting decision-making policies over time based on feedback from dynamic environments [4,6]. Furthermore, RL does not rely on precise mathematical models [4], making it more suitable for real-world scenarios where power generation dynamics may be uncertain or challenging to accurately model.

However, despite the potential of RL-based models to offer improved power scheduling solutions, only a few studies (such as [4,5,6,7,8,9,10]) are available in the literature. These RL-based models tend to prioritize economic costs and overlook environmental impacts, leading to excessive carbon emissions and neglecting long-term consequences [6]. Additionally, scalability remains a challenge for both the model- and RL-based approaches, as the dimensionality grows exponentially with an increasing number of units [11]. Consequently, simplified power scheduling definitions are often used, which may not adequately represent realistic power systems.

This study aims to address these limitations by proposing a novel deep RL-based method for power scheduling that minimizes both economic and environmental costs. The algorithm utilizes an agent-based contextual simulation environment, where generating units are represented as agents. The simulation environment automatically corrects illegitimate commitments and adjusts supply capacity to meet demand, enabling agents to learn optimal behaviors more efficiently. Furthermore, the proposed method mitigates the dimensionality problem associated with large-scale problems, distinguishing it from existing approaches in the literature. The power scheduling dynamics are simulated using a Markov decision process (MDP), and the results are fed into a deep Q-network (DQN) with separate output nodes (ON and OFF) for each agent, allowing for effective decision making. The remainder of this article is organized as follows: The description of the power scheduling problem is presented in Section 2. Then, Section 3 provides the technical details of the proposed methodology, which is demonstrated with a numerical example in Section 4. Concluding remarks follow in the Section 5.

2. Problem Description

Given a power system with $n$ generating units and a power scheduling horizon of 24 h, let $z_{it}\in \left\{\mathrm{0,1}\right\}$ denote the commitment (ON/OFF) status of unit $i$ at period $t$, and $p_{it}\in [0,\infty )$ be the optimal power output of unit $i$ at period $t$ (MW).

2.1. Objective Function

The operating cost of power production at each period $t$ is often defined by the sum of the production costs ($c_{it}^{prod}$), start-up costs ($c_{it}^{ON}$), and shutdown costs ($c_i^{OFF}$) of all units. This definition of total operation cost ignores environmental constraints, which depend on local regulation and emission allowance trading market schemes [12]. A properly represented power scheduling model should also include other costs that are not linked to fuel prices but related to fuel consumption and technological efficiency [3]. Hence, it is necessary to include emission costs ($c_{it}^{emis}$) as part of the total operation cost [13]. In the existing models, the environmental impact of power generation is primarily addressed in the form of emissions constraints and penalties. Emission constraints involve setting limits on the maximum allowable GHGs and other pollutants, whereas the penalty method assigns penalty costs to units emitting beyond the allowed limits. However, a common issue with both the emission constraint and penalty methods is their lack of flexibility, as they do not account for dynamic adjustments based on real-time changes, such as demand fluctuations, outages of units, and fuel and other operational expenses. Both methods tend to prioritize compliance with emission limits rather than actively pursuing emission reduction strategies. To address these limitations, this study proposes the adoption of emission cost parameters integrated into the main objective function. By representing the emissions produced per MW for different types of units, this approach offers a continuous and gradual representation of environmental impacts [13]. This not only allows for more nuanced and flexible decision making by treating emissions as a continuous variable rather than a binary constraint, but it also enables the assessment of the emission-reduction potential of different power plants.

Thus, the objective function representing the total operational economic and environmental costs of the entire planning horizon ($\mathcal{C}$) can be expressed as Equation (1):

$$\mathcal{C}=\sum _{t=1}^{24}\sum _{i=1}^n\left\{z_{it}(c_{it}^{prod}+c_{it}^{emis})+z_{it}\left(1-z_{i,t-1}\right)c_{it}^{ON}+\left(1-z_{it}\right)z_{i,t-1}{c}_i^{OFF}\right\}$$

(1)

where

$$c_{it}^{prod}={\alpha }_i{p}_{it}^2+{\beta }_i{p}_{it}+{\delta }_i$$

(2)

$$c_{it}^{emis}=p_{it}\sum _{h=1}^m{\varphi }_h{\psi }_{ih}$$

(3)

$$c_{it}^{ON}=\left\{\begin{array}{ll}{c}_i^{hot,ON},& {\mathcal{t}}_{i\mathrm{*}}^{down}\le {\mathcal{t}}_{i,t-1}^{OFF}\le {\mathcal{t}}_{i\mathrm{*}}^{down}+{\mathcal{t}}_{i\mathrm{*}}^{cold}\\ c_i^{cold,ON},& {\mathcal{t}}_{i,t-1}^{OFF}>{\mathcal{t}}_{i\mathrm{*}}^{down}+{\mathcal{t}}_{i\mathrm{*}}^{cold}\end{array}\right.$$

(4)

Equation (2) is the production cost function of unit $i$ at period $t$ where ${\alpha }_i$, ${\beta }_i$, and ${\delta }_i$ are the corresponding quadratic, linear, and constant coefficients, respectively. Next, the emission cost of $m$ types of pollutants released by unit $i$ is represented in Equation (3). Since three emission types (namely, $\mathrm{C}{\mathrm{O}}_2$, $\mathrm{S}{\mathrm{O}}_2$, and $\mathrm{N}{\mathrm{O}}_{\mathrm{x}}$) have been considered, $m=3$ in this study. The emissions level is often directly related to the fuel consumption and technological efficiency [3]. As a result, the cost of emissions can be expressed as a linear function of the power outputs [13], where ${\varphi }_h$ is the external cost of emission type $h$ ($/g), and ${\psi }_{ih}$ is the $h^{\mathrm{t}\mathrm{h}}$ emission factor of unit $i$ (g/MW). Finally, the start-up cost given in Equation (2) is a function of the time duration for which unit $i$ has been continuously offline (OFF) until the period $t$, ${\mathcal{t}}_{i,t-1}^{OFF}$. The shutdown costs are fixed but usually negligible [14] and mostly considered zero [3].

2.2. Constraints

The objective function in Equation (1) is required to be minimized subject to different unit-specific and system level constraints as presented in Equations (5)–(9):

$$\mathrm{C}\mathrm{a}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{e}\mathrm{s}:{z_{it}p}_{i^{*}}^{min}\le p_{it}\le z_{it}{p}_{i^{*}}^{max}$$

(5)

$$\mathrm{R}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s}:z_{i,t-1}{z}_{it}\left(p_{i,t-1}-p_{i^{*}}^{down}\right)\le p_{it}\le z_{i,t-1}{z}_{it}\left(p_{i,t-1}+p_{i^{*}}^{up}\right)$$

(6)

$$\mathrm{O}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{s}:{\mathcal{t}}_{it}^{ON}\ge {\mathcal{t}}_{i^{*}}^{up}\mathrm{and}{\mathcal{t}}_{it}^{OFF}\ge {\mathcal{t}}_{i^{*}}^{down}$$

(7)

$$\mathrm{L}\mathrm{o}\mathrm{a}\mathrm{d}\mathrm{B}\mathrm{a}\mathrm{l}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}:\sum _{i=1}^n{z_{it}p}_{it}=d_t$$

(8)

$$\mathrm{R}\mathrm{e}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{e}:\sum _{i=1}^n{z}_{it}{p}_{it}^{max}\ge (1+\mathcal{r})d_t$$

(9)

The unit constraints in Equations (5)–(7) affect each unit taken separately, which collectively accounts for the technical specifications of generating units, and the system constraints in Equations (8) and (9) are used to balance the power supply and demand during each period. Due to the non-convexity of the objective function, the combinatorial nature of commitments, and the time-dependent technical characteristics of the power supply units, solving the objective function in Equation (1) subject to the constraints in Equations (5)–(9) using classical methods is highly computationally demanding even for a moderate number of units and might also stack in some local optima. As a result, the power scheduling problem remains a strongly NP-hard problem, causing the curse of dimensionality, and the implicit burden of computation has limited the scope of numerical optimization [3]. A model-free RL approach may provide a promising methodological framework for solving the power scheduling problem [5].

3. Proposed Methodological Framework

A novel multi-agent deep contextual RL algorithm for power scheduling is proposed by constructing a specialized environment called an “agent-based contextual simulation environment”, whose main peculiarities are explained in Section 3.1. Within the environment, the generating units are represented as cooperative types of RL agents [15]. The agents are active in observing the contextual changes in the environment, and they can make independent decisions regarding their commitment status and optimal power outputs. On the other hand, the agents collaborate to satisfy demand, including the reserve availability at each hour (timestep) described in Equations (8) and (9), and the total operation cost of the entire planning horizon (episode) is to be minimized. The power scheduling dynamics from the agent-to-environment interactions follow an agent-based simulation strategy [16] because the agents are heterogeneous, active, and autonomous. These dynamics are simulated in the form of a Markov decision process (MDP), whose key elements are defined below, and then fed as inputs for the deep RL model.

3.1. The Power Scheduling Dynamics as an MDP

Since the planning horizon is an hourly divided day, each hour is considered a timestep $t,\forall t$. For each timestep $t$ of an episode, the system will be state ${\mathcal{s}}_t$ consisting of different components: current timestep $t$, minimum capacities ($p_{it}^{min};\forall i$), and maximum capacities ($p_{it}^{max};\forall i$) based on the maximum ramp-down ($p_{i*}^{down};\forall i$) and ramp-up ($p_{i*}^{up};\forall i$) rates of the units, current operating (online/offline) time durations $(t_{it};\forall i)$ of the units based on the minimum up-time duration $({\mathcal{t}}_{i*}^{up};\forall i$) and down-time duration $({\mathcal{t}}_{i*}^{down};\forall i$) of the units, and the demand ($d_t$) to be satisfied. The state ${\mathcal{s}}_t$ at timestep $t$ is defined as ${\mathcal{s}}_t=(t,{\mathit{p}}_t^{min},{\mathit{p}}_t^{max},{\mathcal{t}}_t,d_t)$ where $t$ is the current timestep, ${\mathit{p}}_t^{min}$ is a vector of minimum capacities, ${\mathit{p}}_t^{max}$ is a vector of maximum capacities, ${\mathcal{t}}_t$ is a vector of current (online/offline) time duration, and $d_t$ is the demand. Overall, the system’s state space for the entire episode can be described as $\mathcal{S}=\left(\mathit{t},{\mathit{P}}^{min},{\mathit{P}}^{max},\mathit{T},\mathit{d}\right)$. There are two possible actions (switch-to/stay ON or switch-to/stay OFF) for each of the $n$ agents. This implies, there are a total of $2^n$ action combinations (i.e., unit commitments) in the action space $\mathcal{A}$. Thus, each agent $i$ will decide their optimal action (${\mathcal{a}}_{it}\in \left\{\mathrm{0,1}\right\}$) at timestep $t$ (or in state ${\mathcal{s}}_t$). Then, the decisions of all the $n$ agents together constitute an $n$-dimensional vector ${\mathcal{a}}_t={\left\{\mathrm{0,1}\right\}}^n\in \mathcal{A}$. This actions vector ${\mathcal{a}}_t$ is the change of the switch (ON/OFF) status ${\mathit{z}}_t$ of units at period $t$ to ${\mathit{z}}_{t+1}$ at the next period $t+1$. Once the agents take actions ${\mathcal{a}}_t\in \mathcal{A}$ in the current state ${\mathcal{s}}_t\in \mathcal{S}$, there is a transition (or probability) function $\mathcal{P}\left({\mathcal{s}}_{t+1}|{\mathcal{s}}_t,{\mathcal{a}}_t\right)$ that leads to the next state ${\mathcal{s}}_{t+1}$. The transition function must satisfy all the constraints.

At each timestep $t$ of the planning horizon, the agents first observe the state ${\mathcal{s}}_t=\left(t,{\mathit{p}}_t^{min},{\mathit{p}}_t^{max},{\mathcal{t}}_t,{\mathit{z}}_t,d_t\right)\in \mathcal{S}$. Then, each agent can decide to be either ON or OFF, which would result in $2^n$ combinations of unit commitment found in an action space $\mathcal{A}$. The decisions of all agents constitute the action vector ${\mathcal{a}}_t={\left\{\mathrm{0,1}\right\}}^n\in \mathcal{A}$. Then, the agents get an aggregate reward $r_t\in \mathbb{R}$ that can lead to the next state ${\mathcal{s}}_{t+1}\in \mathcal{S}$ through a transition (probability) function $\mathcal{P}\left({\mathcal{s}}_{t+1}={\mathcal{s}}^{\prime }|{\mathcal{s}}_t=\mathcal{s},{\mathcal{a}}_t\right)$. Therefore, these power scheduling dynamics can be represented as a 4-tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathit{r})$ Markovian decision process where $\mathcal{S}$ is a state space, $\mathcal{A}$ is an action space, $\mathcal{P}$ is a transition (or probability) function, and $\mathit{r}$ is a reward.

3.2. Agent-Based Contextual Simulation Environment Algorithm

The structure of agent-based simulation environment is roughly similar to an OpenAI Gym as described below.

Step 1. The parameters of supply units are initialized as ${\mathcal{s}}_0=(0,{\mathit{p}}_0^{min},{\mathit{p}}_0^{max},{\mathcal{t}}_0,{\mathit{z}}_0,d_0)$.

Step 2. The minimum and maximum marginal costs of all agents are determined based on the average production cost, Equation (10).

$${\lambda }_i^{min}={\alpha }_i{p}_{i*}^{max}+{\beta }_i+\frac{{\delta }_i}{p_{i*}^{max}}\mathrm{and}{\lambda }_i^{max}={\alpha }_i{p}_{i*}^{min}+{\beta }_i+\frac{{\delta }_i}{p_{i*}^{min}};\forall i$$

(10)

Step 3. The must-ON ($u_{it}^1$) or must-OFF ($u_{it}^0)$ agents are identified based on the operating times, Equation (11).

$$u_{it}^1=1\mathrm{if}0<{\mathcal{t}}_{it}<{\mathcal{t}}_{i*}^{up};\mathrm{and}{u}_{it}^0=1\mathrm{if}-{\mathcal{t}}_{i*}^{down}<{\mathcal{t}}_{it}<0;\forall i.$$

(11)

Step 4. The agents execute their action ${\mathcal{a}}_t$ in state ${\mathcal{s}}_t$, and then pass to the next state ${\mathcal{s}}_{t+1}$ through a transition (or probability) function, $\mathcal{P}\left({\mathcal{s}}_{t+1}|{\mathcal{s}}_t,{\mathcal{a}}_t\right)$, satisfying all constraints.

Step 4.1. The legality of action ${\mathcal{a}}_{it}\in {\mathcal{a}}_t$ of each agent is confirmed and legalized if there are any violations of the constraints specified in Equation (11), as shown in Equation (12).

$${\mathcal{a}}_{it}=1\mathrm{if}{\mathcal{a}}_{it}=0|u_{it}^1=1;\mathrm{and}{\mathcal{a}}_{it}=0\mathrm{if}{\mathcal{a}}_{it}=1|u_{it}^0=0,\forall i.$$

(12)

Step 4.2. The aggregate supply capacity of agents is checked for sufficiency in satisfying the demand and future demands when OFF units have not completed their downtime. Then contextual capacity adjustments are made if necessary, and if possible.

▪

If $\sum _{i=1}^n{\mathcal{a}}_{it}{p}_{it}^{max}<\left(1+\mathcal{r}\right)d_t$, then set each ${\mathcal{a}}_{it}=1|u_{it}^1=0$ based on the increasing order of ${\lambda }_i^{min}$’s of Equation (10) until $\sum _{i=1}^n{\mathcal{a}}_{it}{p}_{it}^{max}\ge \left(1+\mathcal{r}\right)d_t$. If the capacity shortage is not fully corrected due to unconstrained OFF units, then ${\mathcal{s}}_t$ is labeled as a terminal state (${\mathcal{s}}_t^{+}$) that would result an incomplete episode (${\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}=1$).

▪

If $\sum _{i=1}^n{\mathcal{a}}_{it}{p}_{it}^{min}>\left(1+\mathcal{r}\right)d_t$, then set each ${\mathcal{a}}_{it}=0|u_{it}^0=1$ as per the decreasing order of ${\lambda }_i^{min}$’s of Equation (10) until $\sum _{i=1}^n{\mathcal{a}}_{it}{p}_{it}^{min}\le \left(1+\mathcal{r}\right)d_t$. If the excess capacity is not yet fully adjusted due to an insufficient number of unconstrained ON units, it results in an incomplete episode (${\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}=1$) as the state ${\mathcal{s}}_t$ is terminal (${\mathcal{s}}_t^{+}$).

▪

If the current capacity does not satisfy future demands, set each ${\mathcal{a}}_{it}=1\left|\right({\mathcal{t}}_{it}\ge {\mathcal{t}}_{i\mathrm{*}}^{up})$ as per the decreasing order of ${\lambda }_i^{min}$’s of Equation (10). The current state ${\mathcal{s}}_t$ is also labeled as terminal (${\mathcal{s}}_t^{+}$) if the future demands cannot be meet while the offline units must still be OFF due to an insufficient number of unconstrained OFF units.

Step 5. The total operation cost at timestep $t$ is determined. First, start-up and shutdown costs are obtained based on the action ${\mathcal{a}}_t$. Second, a lambda iteration algorithm is used for solving the optimal power loads $p_{it}$, in Equation (2), which are then used to estimate the emission costs specified in Equation (3). Lastly, the total operation cost is obtained using Equation (13) where $z_{i,t+1}={\mathcal{a}}_{it},\forall i$.

$${\mathcal{C}}_t=\sum _{i=1}^n\left\{z_{i,t+1}(c_{it}^{prod}+c_{it}^{emis})+z_{i,t+1}\left(1-z_{it}\right)c_{it}^{ON}+\left(1-z_{i,t+1}\right)z_{it}{c}_i^{OFF}\right\}$$

(13)

Step 6. The agents get an aggregate reward according to the predefined function given in Equation (14), which is the negative of the normalized total operation cost scaled to 100.

$$r_t=\mathcal{R}\left({\mathcal{s}}_t,{\mathcal{a}}_t,{\mathcal{s}}_{t+1}\right)=\left(1-\frac{(1-{\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}){\mathcal{C}}_t+{\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}{\mathcal{C}}_t^{+}-{\mathcal{C}}^{min}}{{\mathcal{C}}^{max}-{\mathcal{C}}^{min}}\right)\times 100$$

(14)

In the episodic task of RL, incomplete episodes need to be avoided, and large penalties are recommended by [11]. For this purpose, while the cost function in Equation (13) is used for non-terminal states, the cost for terminal states is defined as ${\mathcal{C}}_t^{+}={\mathcal{C}}^{max}-\frac{t}{23}({\mathcal{C}}^{max}-{\mathcal{C}}^{p^{max}})$ where ${\mathcal{C}}^{max}=\sum _{i=1}^n{\lambda }_i^{max}{p}_{i\mathrm{*}}^{max}$ and ${\mathcal{C}}^{p^{max}}$ is the sum of Equations (2) and (3), assuming $p_{it}=p_{i\mathrm{*}}^{max}$. This provides evenly distributed penalties, maintaining the desired proximity to the final timestep.

Step 7. If the current state is terminal (i.e., ${\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}=1$) or $t\ge 24$, then go to Step 1 to re-initialize the environment and restart a new episode to state ${\mathcal{s}}_0$. But, if $t<24$ and ${\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}=0$, the agents pass to the next state ${\mathcal{s}}_{t+1}=(t+1,{\mathit{p}}_{t+1}^{min},{\mathit{p}}_{t+1}^{max},{\mathcal{t}}_{t+1},{\mathit{z}}_{t+1},d_{t+1})$ where ${\mathit{z}}_{t+1}={\mathcal{a}}_t$ is a vector commitment status; ${\mathit{p}}_{t+1}^{min}=\mathit{max}\left\{{\mathit{p}}_{\mathrm{*}}^{min},{\mathit{z}}_t^{\prime }{\mathit{z}}_{t+1}\left({\mathit{p}}_t-{\mathit{p}}_{\mathrm{*}}^{down}\right)\right\}$ is a vector of minimum capacities; ${\mathit{p}}_{t+1}^{max}=\mathit{min}\left\{{\mathit{p}}_{\mathrm{*}}^{max},{\mathit{z}}_t^{\prime }{\mathit{z}}_{t+1}\left({\mathit{p}}_t+{\mathit{p}}_{\mathrm{*}}^{up}\right)+{\mathbb{I}}_{[{\mathit{z}}_t^{\prime }{\mathit{z}}_{t+1}=0]}{\mathit{p}}_{\mathrm{*}}^{max}\right\}$ is a vector of maximum capacities; ${\mathcal{t}}_{t+1}=({\mathcal{t}}_t+1$ if ${{\mathbf{z}}_{t+1}=1|\mathcal{t}}_t>0$, $-1$ else if ${{\mathbf{z}}_{t+1}=0|\mathcal{t}}_t>0$, $1$ else if ${{\mathbf{z}}_{t+1}=1|\mathcal{t}}_t<0$, ${\mathcal{t}}_t-1$ if ${{\mathbf{z}}_{t+1}=0|\mathcal{t}}_t<0)$ is a vector of operating time durations.

Step 8. Update the must-ON and must-OFF agents for the next timestep $t+1$ as defined in Equation (11): $u_{i,t+1}^1=$1 if $0<{\mathcal{t}}_{i,t+1}<{\mathcal{t}}_{i\mathrm{*}}^{up}$, and $u_{i,t+1}^0=$1 if $-{\mathcal{t}}_{i\mathrm{*}}^{down}<{\mathcal{t}}_{i,t+1}<0;\forall i$.

Step 9. The execution of ${\mathcal{a}}_t$ in the environment returns the next state (${\mathcal{s}}_{t+1}$), the reward ($r_t$), indicates whether the state is terminal (${\mathbb{I}}_{\left[{\mathcal{s}}_t^{+}\right]}$) and loads dispatch information together with the legally confirmed action (${\mathcal{a}}_t,{\mathit{p}}_t)$.

It should be noted that action ${\mathcal{a}}_t$ returned in Step 9 is not necessarily the same as the action ${\mathcal{a}}_t$ executed in Step 4 since the environment makes contextual corrections in Step 4.1 and 4.2 using the idea of a contextual search [17]. This agent-based contextual simulation environment, one of the main contributions of this study, can be highly effective in reducing the computing and training time of the multi-agent deep contextual RL model described below.

3.3. Deep Contextual Reinforcement Learning

At each timestep $t\in \mathcal{T}$, the agents observe a state ${\mathcal{s}}_t$ from $\mathcal{S}$ and select their respective actions ${\mathcal{a}}_t$ from the action space $\mathcal{A}$ according to a policy $\pi \left({\mathcal{a}}_t\right|{\mathcal{s}}_t)$, where $\pi$ is a mapping from states ${\mathcal{s}}_t$ to actions ${\mathcal{a}}_t$. The agents then receive a reward $r_t$ and proceed to the next state ${\mathcal{s}}_{t+1}$. This process continues until the agents finish the entire episode or reach a terminal state, both of which reinitialize the environment. The agents’ goal is to learn a policy $\pi \left({\mathcal{a}}_t|{\mathcal{s}}_t\right)$ that maximizes the long-run cumulative sum of rewards called return, defined as $G_t\doteq \sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{r}_{t+k}$ where $\gamma$ is a discount rate $\gamma \in \left[\mathrm{0,1}\right]$ of the MDP. The expected return of action, ${\mathcal{a}}_t$ in the state ${\mathcal{s}}_t$ can be expressed as an action-value function $Q^{\pi }\left({\mathcal{s}}_t,{\mathcal{a}}_t\right)=\mathbb{E}\left(G_t\right|{\mathcal{s}}_t,{\mathcal{a}}_t)$. It can be approximated using a deep Q-network (DQN) which can be applied in a high-dimensional state and/or action space [18]. The action-value function can now be written as $Q({\mathcal{s}}_t,{\mathcal{a}}_t|\mathit{\theta })$, where $\mathit{\theta }$ consists of the parameters of the DQN model whose inputs are the power scheduling dynamics simulated from the environment in the form of MDPs. The size of action space $\mathcal{A}$ is $2^n$, which may render an exponential growth in computation. Thus, it is impractical to parameterize the model into $2^n$ output nodes. Instead, it is parameterized into $2n$ output nodes corresponding to the two possible actions (ON/OFF) of each agent. As a result, the model estimates action-values for $2n$ output nodes, and then the decisions of agents made using an exponential decay epsilon-greedy exploration strategy collectively constitute the action vector ${\mathcal{a}}_t$.

In a power scheduling problem, the action in a particular state affects the rewards and a set of future states, which yields serial correlations among the MDPs. In such cases, direct application of DQN may not be efficient, as it might result in unstable and slow learning processes [11]. Employing the notion of experience replay, the autocorrelation among the states may be properly addressed, and the training process can be expedited and stabilized [19]. After storing a transition tuple (${\mathcal{s}}_t,{\mathcal{a}}_t,r_t,{\mathcal{s}}_{t+1})$ to a replay buffer $\mathbb{B}$, a batch $b$ of experiences $\left(\mathcal{s},\mathcal{a},\mathit{r},{\mathcal{s}}^{\prime }\right)$ is sampled to approximate the action-value function $Q\left(\mathcal{s},\mathcal{a}|\mathit{\theta }\right)$ and the target network $Q\left({\mathcal{s}}^{\prime },{\mathcal{a}}^{\prime }|{\mathit{\theta }}^{\prime }\right)$, where $\mathcal{s}$ and ${\mathcal{s}}^{\prime }$ are of size $(b\times 2n)$, and the sizes of $\mathcal{a}$ and $\mathit{r}$ are $(b\times n)$ and $(b\times 1)$, respectively. The DQN algorithm minimizes the mean-squared loss (i.e., temporal difference) $L_e\left(\mathit{\theta }\right)$ defined by

$$L_e\left(\mathit{\theta }\right)={\mathbb{E}}_{\left(\mathcal{s},\mathcal{a},\mathit{r},{\mathcal{s}}^{\prime }\right)~\mathbb{B}}{\left[\mathit{r}+\gamma \underset{{\mathit{a}}^{\prime }}{\mathit{max}}Q\left({\mathcal{s}}^{\prime },{\mathcal{a}}^{\prime }|{\mathit{\theta }}^{\prime }\right)-Q\left(\mathcal{s},\mathcal{a}|\mathit{\theta }\right)\right]}^2$$

(15)

where ${\mathit{\theta }}^{\prime }$ is the parameter of target network, which is periodically updated from the Q-network parameters $\mathit{\theta }$, and $e$ denotes the iteration index.

4. Demonstrative Example

The applicability of the proposed deep contextual RL method is demonstrated with the power system investigated in [20]. The test system consists of five units, for which supply and demand profiles and emission parameters can be found in [20]. The deep RL utilizes a feedforward neural network featuring a rectified linear unit (ReLU) activation function on both the hidden and output layers. The learning rate and discount factor were set to 0.0001 and 0.99, respectively, and the Adam’s optimizer was used. Employing the popular genetic algorithm, the optimal cost in [20] was $430,331, summing up the start-up, production, and emission costs of $3140 (0.7%), $289,178 (67.2%), and $138,010 (32.1%), respectively. On the other hand, the proposed method in this study yields an improved optimal operation cost of $413,122 as shown in

Table 1. Optimal commitments, optimal loads, and available reserve of test power system I using the proposed method.

Hour $\left(\mathit{t}\right)$	Optimal Commitments					Optimal Loads (MW)					$\mathcal{r}$ (%)
	${\mathit{z}}_{1\mathit{t}}$	${\mathit{z}}_{2\mathit{t}}$	${\mathit{z}}_{3\mathit{t}}$	${\mathit{z}}_{4\mathit{t}}$	${\mathit{z}}_{5\mathit{t}}$	${\mathit{p}}_{1\mathit{t}}$	${\mathit{p}}_{2\mathit{t}}$	${\mathit{p}}_{3\mathit{t}}$	${\mathit{p}}_{4\mathit{t}}$	${\mathit{p}}_{5\mathit{t}}$
1	1	0	0	0	0	400.0	0	0	0	0	13.8
2	1	0	1	0	0	426.5	0	23.5	0	0	30.0
3	1	0	1	0	0	450.9	0	29.1	0	0	21.9
4	1	0	1	0	0	455.0	0	45.0	0	0	17.0
5	1	0	1	0	0	455.0	0	75.0	0	0	10.4
6	1	1	1	0	0	455.0	36.6	58.4	0	0	30.0
7	1	1	1	0	0	455.0	52.0	73.0	0	0	23.3
8	1	1	1	0	0	455.0	62.3	82.7	0	0	19.2
9	1	1	1	0	0	455.0	72.6	92.4	0	0	15.3
10	1	1	1	1	0	455.0	77.7	97.3	20	0	22.3
11	1	1	1	1	0	455.0	93.1	111.9	20	0	16.9
12	1	1	1	1	0	455.0	103.3	121.7	20	0	13.6
13	1	1	1	1	0	455.0	77.7	97.3	20	0	22.3
14	1	1	1	0	0	455.0	72.5	92.5	0	0	15.3
15	1	1	1	0	0	455.0	62.3	82.7	0	0	19.2
16	1	1	1	0	0	455.0	36.6	58.4	0	0	30.0
17	1	0	1	0	0	455.0	0	45.0	0	0	17.0
18	1	0	1	1	0	455.0	0	75.0	20	0	20.9
19	1	0	1	1	0	455.0	0	125.0	20	0	10.8
20	1	0	1	1	1	455.0	0	130.0	55	10	10.8
21	1	0	1	1	0	455.0	0	125.0	20	0	10.8
22	1	0	1	1	0	455.0	0	75.0	20	0	20.9
23	1	0	1	0	0	455.0	0	45.0	0	0	17.0
24	1	0	1	0	0	426.4	0	23.6	0	0	30.0

and Figure 1 and Figure 2, corresponding to $16,808 (4.0%) lower daily operation cost than the results reported in [20] as compared in

Table 2. Comparison of the optimal costs (start-up cost, production cost, emission cost, and total cost) of test power system I between genetic algorithm (GA) [20] and the proposed RL method.

Hour $\left(\mathit{t}\right)$	Genetic Algorithm [20]				Proposed RL
	Start-Up	Production	Emission	Total	Start-Up	Production	Emission	Total
1	0	8466	4824	13,290	0	7553	4241	11,793
2	0	8466	4828	13,294	560	9061	4702	14,324
3	60	10,564	5044	15,668	0	9560	5004	14,564
4	0	10,564	5044	15,608	0	9893	5170	15,063
5	1120	11,327	5825	18,271	0	10,395	5401	15,797
6	0	11,327	5825	17,151	1100	11,427	5555	18,082
7	0	11,327	5825	17,151	0	11,930	5786	17,717
8	60	13,425	6045	19,529	0	12,267	5940	18,207
9	0	13,425	6045	19,469	0	12,604	6094	18,699
10	340	13,523	6145	20,008	340	13,591	6251	20,183
11	30	15,621	6365	22,016	0	14,099	6483	20,582
12	0	15,621	6365	21,986	0	14,439	6637	21,076
13	0	13,523	6145	19,668	0	13,591	6251	19,843
14	30	13,425	6045	19,499	0	12,604	6094	18,699
15	1100	13,456	6045	20,601	0	12,267	5940	18,207
16	0	11,358	5825	17,182	0	11,427	5555	16,982
17	0	11,358	5825	17,182	0	9893	5170	15,063
18	0	11,358	5825	17,182	170	11,213	5481	16,865
19	340	13,554	6145	20,039	0	12,059	5866	17,926
20	0	13,554	6145	19,699	60	13,862	6085	20,007
21	0	13,554	6145	19,699	0	12,059	5866	17,926
22	0	11,358	5825	17,182	0	11,213	5481	16,695
23	60	10,564	5044	15,668	0	9893	5170	15,063
24	0	8466	4824	13,290	0	9061	4702	13,764
Total	3140	289,178	138,013	430,331	2230	275,962	134,931	413,122

and Figure 3. The total cost is composed of $2230 (0.5%) for start-up costs, $275,962 (66.8%) for production costs, and $134,931 (32.6%) for emission costs. It is also asserted that the proposed method may be computationally efficient by adopting the experience replay. The scalability of the proposed method may thus be tested with a large-scale power system comprising a large number of generating units. Duplicating the five-unit test system multiple times and scaling the demands proportionately, the proposed method has been applied to obtain the optimal power scheduling scheme of individual units. The optimal costs of duplicated large-scale power systems are summarized in

Table 3. Optimal costs of power production for large-scale systems.

Number of Units	Cost ($)
	Start-Up	Production	Emission	Total
10	4840	545,837	270,724	821,401
20	9300	1,083,979	540,476	1,633,754
30	14,370	1,622,473	811,866	2,448,710
40	18,980	2,160,114	1,082,666	3,261,760
50	31,660	2,744,090	1,329,071	4,104,821
80	51,320	4,369,093	2,129,526	6,549,939
100	58,960	5,456,700	2,674,207	8,189,867

. It is worth noting that the optimal operating cost of each test system is lower than the scaled optimal operating cost of the original five-unit system. It is implied that the proposed method may easily be extended to render an economically and environmentally better solution for larger-scale power systems.

5. Concluding Remarks

This article presents a novel RL-based algorithm designed to optimize the economic and environmental cost of power scheduling. The algorithm utilizes a contextually corrective agent-based RL environment, which simulates power scheduling dynamics using the framework of MDP. To evaluate the applicability and performance of the proposed method, the algorithm is tested on different test systems comprising up to 100 generating units. It is demonstrated that the algorithm provides superior solutions and is scalable to handle larger power systems. The potential for incorporating renewable power sources and investigating their impacts further highlights the versatility and applicability of the proposed method in addressing real-world power scheduling challenges.