Novel Energy Trading System Based on Deep-Reinforcement Learning in Microgrids

Abstract

Inefficiencies in energy trading systems of microgrids are mainly caused by uncertainty in non-stationary operating environments. The problem of uncertainty can be mitigated by analyzing patterns of primary operation parameters and their corresponding actions. In this paper, a novel energy trading system based on a double deep Q-networks (DDQN) algorithm and a double Kelly strategy is proposed for improving profits while reducing dependence on the main grid in the microgrid systems. The DDQN algorithm is proposed in order to select optimized action for improving energy transactions. Additionally, the double Kelly strategy is employed to control the microgrid’s energy trading quantity for producing long-term profits. From the simulation results, it is confirmed that the proposed strategies can achieve a significant improvement in the total profits and independence from the main grid via optimized energy transactions.

1. Introduction

Recently, microgrid (MG) systems have been widely established for distributed power grids [1,2]. Compared with conventional grid systems, energy transmission losses and carbon emissions can be minimized by the MG [3]. The MG has been typically designed for the self-sufficiency of energy by operating energy management and transactions independently from the main grid [4,5,6]. Therefore, the MG can relieve disasters such as cyber-physical attacks and power outages [4]. In an energy trading system (ETS), the MG acts as a prosumer and interacts to maximize social welfare through a distributed power system of an electricity trade model [6,7]. To operate off-grid, the ETS requires the forecasting of demand and supply requirements and the balancing of the possession of energy [5]. In addition, profits should be guaranteed to prove the economic feasibility of the ETS.

Several studies have been conducted for a variety of operating environments on the energy independence and profitability of the MG system [8,9,10,11]. In [8], multi-MG interconnection and policies have been proposed for independent energy management, which eventually saves social energy trading costs and improves reliability. An optimized strategy for each MG has been recommended for an efficient grid-to-grid (G2G)-based energy trading method [9]. In [10], the interconnected multi-MG energy trading model has been proposed for the optimal energy scheduling of MGs by improving the energy efficiency of the system. Independence of the energy from the main grid can be achieved via interdependence between developed and developing MGs when the G2G-based energy trade is formed [11]. In [12], temporal complementarity between supply and demand in energy trade has been proposed for the entire community of energy self-sufficiency. For an efficient ETS, fluent switching between consumer and seller can be a crucial factor for variant operating situations. Another important aspect of the ETS is improving profits in a balanced way. In order to improve the profits via energy trading, it is necessary to predict and utilize information about each MG. In order to predict energy surplus or deficit, the ETS can be modeled as a resilience system [13]. For maximizing the benefits to consumers, optimal transacted quantities have been determined by considering uncertainties and coordinating day-ahead optimal scheduling [14]. In addition, several algorithms have been proposed to manage the MG for profits and balanced social welfare [15]. A compute clearing price and volume (CCPV) algorithm has been introduced to maximize the equilibrium quantity of energy trading for maximizing profits [15]. Additionally, an envy-free division (EFD) algorithm has been proposed to increase social welfare by allowing agents to equally share opportunities for profits [15]. In [16], a multi-bilateral economic dispatch formulation has been proposed based on optimal power flow to maximize community welfare while avoiding critical grid conditions.

In order to optimize energy trading, several schemes have been proposed and analyzed [17,18,19,20]. Reinforcement learning (RL) based algorithms have been proposed for the improvement in decision-making for the energy management of MG [17,18,19,20]. For operating any bounded utility of stochastic character, a linear-inaction scheme has been proposed to maximize average revenue through the incorporation of the RL algorithm and game theory [18]. The RL algorithm has been used to reduce the supply-demand mismatch problem for optimizing the energy-independent model [19]. Considering practical energy trading environments, the MG energy management has been modeled by the deep-RL algorithm for solving the problem of random variations in both supply and demand [20]. However, the presented systems have generally been focused on short-term efficiency for constructing a real-time trading system without considering the long-term perspective. Furthermore, due to simulation in stationary environments, they have applicability limitations to the realistic ETS.

In this paper, a novel G2G-based ETS is proposed for a self-sufficient energy system which can manage independence on the main grid as well as profits by lower trading costs [11,21]. Utilizing a double Kelly strategy, the optimized quantity of energy trading can be computed in the uncertainty of the future for each MG. The ETS model is optimized by a double deep Q-networks (DDQN) to improve the performance of the ETS. Therefore, compared with the conventional approaches, contributions of the proposed system can be summarized in the following three aspects:

• Considering the quantity of energy trading, conventional studies have been largely determined by static trading systems. Unlike conventional studies, we employed the double Kelly strategy, considering dynamic determinations and flexible energy transactions. The proposed ETS can improve the utility of energy resources and the activation of energy transactions;
• Conventional studies have typically been simulated in stationary environments. However, the proposed system is simulated in non-stationary environments in order to reflect a more realistic market scenario, considering the changeable patterns of the quantity of energy trading, surplus energy, energy prices, energy production by weather, and monthly demand;
• Considering the ETS, it is shown that most of the existing schemes have typically focused on profitability rather than independence. The proposed system has paid attention to both, simultaneously achieving profitability and independence by the DDQN algorithm and the double Kelly strategy.

The rest of this paper is organized as follows. In Section 2, the novel energy trading system model of the MG is introduced. In Section 3, simulation results of the energy transaction are presented, and the performance of the proposed system is analyzed. Finally, conclusions are drawn in Section 4.

2. Novel Energy Trading System Model

The configuration of the G2G-based ETS in this paper is shown in Figure 1. In the MG, solar energy resources are used as the power source. Insufficient energy quantity can be supplied by the main grid. The energy transactions are performed by determining the optimal quantity of energy trading based on the double Kelly strategy. The MGs are sought to reach a consensus by acting to maximize their respective interests and thus benefitting entire system.

2.1. Double Kelly Strategy

The Kelly strategy has been known to pursue long-term profits and prepare for the uncertainty of the future [22]. The double Kelly strategy is a refined version of the Kelly strategy for the activation of the energy trading market. The limitation of analyzing the risky transactions via the current state can be solved by using the double Kelly strategy. As a result, optimal transactions are conducted to minimize risk by considering the loss of the transaction. In addition, the double Kelly strategy sees the transaction proceed flexibly according to the situation, on a case-by-case basis. Applying the double Kelly strategy to the energy trading model, the energy is regarded as capital from the Kelly function [22]. The winning probability is given by

$$0\le p_w\le 1,$$

(1)

where $p_w$ denotes the winning probability. In the proposed ETS, $p_w$ is set by the energy production rate related to weather [23]. The maximum value of $p_w$ can be achieved when energy production is optimized. A profit and loss can be represented by

$$r_W={\mathrm{B}}_{exp}^{\left(t\right)}+1,$$

(2)

$${\mathrm{B}}_{exp}^{\left(t\right)}=S^{\left(t\right)}{p}_{change},$$

(3)

where $r_W$ is compensation for good transactions, ${\mathrm{B}}_{exp}^{\left(t\right)}$ is the expected benefits in energy transactions at time step $t$, $S^{\left(t\right)}$ is stored energy and $p_{change}$ is expected price change. The value of ${\mathrm{B}}_{exp}$ can be found by predicting the price and profits through energy trading using the transition probability of the weather.

Assuming a certain probability of profit and loss with initial energy quantity set to be ${\mathrm{E}}_0$, the energy quantity of ${\mathrm{E}}_1$ after the transaction is given by

$${\mathrm{E}}_1={\mathrm{E}}_0{\left(1+{\mathrm{K}r}_W\right)}^{W_1}{\left(1+{\mathrm{K}r}_L\right)}^{1-W_1},$$

(4)

$${\mathrm{E}}_{1,win}={\mathrm{E}}_0\left(1+{\mathrm{K}r}_W\right),$$

(5)

$${\mathrm{E}}_{1,lose}={\mathrm{E}}_0\left(1+{\mathrm{K}r}_L\right),$$

(6)

where $\mathrm{K}$ is a trading ratio, $W$ is a random variable which is 1 with $p_w$ and 0 with $1-p_w$, and $r_L$ is a loss in energy transactions. The case of ${\mathrm{E}}_2$ after two transactions can be expressed by

$$\begin{gathered} {\mathrm{E}}_2={\mathrm{E}}_1{\left(1+{\mathrm{K}r}_W\right)}^{W_2}{\left(1+{\mathrm{K}r}_L\right)}^{1-W_2} \\ ={\mathrm{E}}_0{\left(1+{\mathrm{K}r}_W\right)}^{W_1+W_2}{\left(1+{\mathrm{K}r}_L\right)}^{2-W_1-W_2}. \end{gathered}$$

(7)

The energy ${\mathrm{E}}_n$ after $n$ transactions can be written as

$${\mathrm{E}}_n={\mathrm{E}}_0{\left(1+{\mathrm{K}r}_W\right)}^{\left({\displaystyle {\sum }_{i=1}^n}{W}_i\right)}{\left(1+{\mathrm{K}r}_L\right)}^{\left(n-{\displaystyle {\sum }_{i=1}^n}{W}_i\right)}.$$

(8)

When $W_{N,win}$ is the number of high performances with $N$ times trade, it can be rewritten as

$${\mathrm{E}}_n={\mathrm{E}}_0{\left(1+{\mathrm{K}r}_W\right)}^{W_{N,win}}{\left(1+{\mathrm{K}r}_L\right)}^{n-W_{N,win}}.$$

(9)

where $W_{N,win}$ is the number of good performances out of a total of $n$ trades. Since $W_{N,win}$ is a variable following a binomial distribution, the average $n{p}_w$ is obtained by applying the law of large numbers under the assumption that $N$ is large enough. Then, the energy ${\mathrm{E}}_n$ after $n$ transactions can be modified by

$${\mathrm{E}}_n={\mathrm{E}}_0{\left(1+{\mathrm{K}r}_W\right)}^{n{p}_w}{\left(1+{\mathrm{K}r}_L\right)}^{n\left(1-p_w\right)},$$

(10)

The ${\mathrm{K}}_{opt}$ at the point where the ${\mathrm{E}}_n/{\mathrm{E}}_0$ value is maximized can be given by

$$\begin{gathered} {\mathrm{K}}_{opt}=\arg \underset{\mathrm{K}}{\max }\left(\frac{{\mathrm{E}}_n}{{\mathrm{E}}_0}\right)=\arg \underset{\mathrm{K}}{\max }(ln\left(\frac{{\mathrm{E}}_n}{{\mathrm{E}}_0}\right)) \\ =\arg \underset{\mathrm{K}}{\max }(ln\left({\left(1+{\mathrm{K}r}_W\right)}^{n{p}_w}{\left(1+{\mathrm{K}r}_L\right)}^{n\left(1-p_w\right)}\right)) \end{gathered}$$

(11)

$$\begin{gathered} \frac{d}{dl}\left(ln({\left(1+{\mathrm{K}r}_W\right)}^{n{p}_w}{\left(1+{\mathrm{K}r}_L\right)}^{n\left(1-p_w\right)})\right)=\frac{n{p}_w{r}_W}{1+{\mathrm{K}r}_W}+\frac{n\left(1-p_w\right)r_L}{1+{\mathrm{K}r}_L} \\ =\frac{n\left(p_w{r}_W+r_L+K{r}_W{r}_L-p_w{r}_L\right)}{\left(1+{\mathrm{K}r}_W\right)\left(1+{\mathrm{K}r}_L\right)}=0, \end{gathered}$$

(12)

$${\mathrm{K}}_{opt}=-\frac{p_w}{r_L}-\frac{1-p_w}{r_W},$$

(13)

In general, since $r_L=-1$, it can be expressed as

$${\mathrm{K}}_{opt}=p_w-\frac{1-p_w}{r_W},$$

(14)

The tradeable energy for transactions which can be derived from the double Kelly strategy can be described by

$${\mathrm{K}}_{double}=2{\mathrm{K}}_{opt},$$

(15)

$$T_i^{\left(t\right)}={\mathrm{K}}_{double}{b}_i^{\left(t\right)},$$

(16)

where ${\mathrm{K}}_{double}$ denotes tradeable energy quantity factor using double Kelly strategy, $T_i^{\left(t\right)}$ is the tradeable quantity of energy of $M{G}_i$ at time step $t$ and $b_i^{\left(t\right)}$ is the surplus energy of $M{G}_i$ at time step $t$. In Equation (16), the long-term profits can be obtained by determining an appropriate quantity of energy trading for each time step.

2.2. Reinforcement Learning for Optimized Energy Trading System

2.2.1. Q-Learning Algorithm

A Q-learning algorithm is employed to build an energy trading system. It is a well-known model-free RL algorithm for solving discrete spatial problems [24]. In the Q-learning algorithm, the optimal policy is to maximize the total reward by successive states. When a specific action $a$ is performed in a certain state $s$ for the Q-learning algorithm, the Q-value is used to determine the value of the action. The Q-value is then determined by the following equations

$$Q_{t+1}\left(s_t,a_t\right)=\left(1-lr\right)Q_t\left(s_t,a_t\right)+lr\left(r_{t+1}+\gamma maxQ\left(s_{t+1},a_{t+1}\right)\right),$$

(17)

$$X_t^Q=r_{t+1}+\gamma maxQ\left(s_{t+1},a_{t+1}\right),$$

(18)

where $lr$ is the learning rate of Q-function, $s_t$ is the state at time $t$, $a_t$ is the action at time step $t$, $r_{t+1}$ is the reward at time step $t+1$ and $\gamma$ is the discount factor. The Q-value converges to the optimal Q-value to derive the optimal policy by repeating Equation (17). The sampling of the target network in Equation (18) is required for convergence.

The Q-learning algorithm has been used to converge optimal policy to improve system performance [25,26]. However, there are a couple of challenging problems with the optimal policy. One is the memory shortage problem [27], and the other is overestimation [28]. Deep Q-networks (DQN) algorithm has been proposed to solve the memory shortage problem [27]. However, the overestimation problem still exists in the DQN algorithm. Therefore, deep-RL with a double Q-learning algorithm named DDQN algorithm has been proposed to solve these two problems simultaneously [28].

2.2.2. DDQN Algorithm

The DDQN algorithm is approximated by adding a neural network to overcome the memory shortage problem; the overestimation problem can be solved by adding an additional neural network. The Q-value in the DDQN algorithm is determined by the following equations

$$Q_{\mathrm{A}+1}\left(s_t,a_t\right)=\left(1-lr\right)Q_A\left(s_t,a_t\right)+lr(r_{t+1}+\gamma Q_B\left(s_{t+1},argmax{Q}_A\left(s_{t+1},a_{t+1}\right)\right),$$

(19)

$${\mathrm{X}}_t^{DDQN}=r_{t+1}+\gamma Q_B\left(s_{t+1},argmax{Q}_A\left(s_{t+1},a_{t+1}\right)\right),$$

(20)

where $Q_A$ and $Q_B$ are primary and target networks, respectively. In Equation (19), the expression $maxQ\left(s_{t+1}, a_{t+1}\right)$ from Equation (17) is divided into selection and evaluation parts [29]. In the decision-making process, the DDQN algorithm is used to select an action and evaluate the selected action. In the DQN algorithm, the selection and evaluation parts are used by only one network. However, the overestimation problem can be caused by the DQN algorithm with only one network. Dividing one network into two networks is one of the methods which can be employed to overcome the overestimation problem. As shown in Equation (20), networks are separated in the DQN algorithm to reach an optimal value. The $Q_A$ and $Q_B$ are used to select actions and to evaluate the Q-value from selected actions, respectively. As the action space increases, the estimation error of the Q-value can be increased in the DQN algorithm. On the other hand, the estimation error of the DDQN algorithm can be underestimated rather than overestimated due to low variance in comparison to the DQN algorithm [28]. Nevertheless, the DDQN algorithm is used for the advantages of error estimation and stable learning in the proposed ETS.

2.3. Proposed Energy System

2.3.1. Data Settings

The monthly consumption patterns and weather transition probabilities have been analyzed to generate reliable one-year data of MG. The average rate of change in consumption, which varies monthly, is shown in

Table 1. The rate of the monthly change in average energy demand.

Month	January	February	March	April	May	June
Monthly demand change ratio	+19%	+19%	−2%	−4%	−14%	−11%
Month	July	August	September	October	November	December
Monthly demand change ratio	−10%	+7%	−2%	−10%	+2%	+6%

[30].

In Figure 2, transition probability by weather is calculated by analyzing 10 years of weather data from the Korea Meteorological Administration. The energy generation efficiency, according to weather state, is averaged and used to create data.

In order to make a more realistic proposed model, the demand, supply, weather, and efficiency of production are considered in the process of data settings. For the reliability of the data, one year’s worth of data is generated with added variance.

2.3.2. Energy Transaction System Function Settings

The energy price is determined by the cost price of MGs and the price ratio. The energy price is given by

$$P_i^{\left(t\right)}=cos{t}_{price}\left(1+p_i\right),$$

(21)

$$p_i=\left(M^{\left(t\right)}+1\right)/\left(\frac{1+b_i^{\left(t\right)}}{W_i^{\left(t\right)}}\right),$$

(22)

where $cos{t}_{price}$ denotes levelized cost of electricity (LCOE), $p_i$ is price ratio of $M{G}_i$, $M^{\left(t\right)}$ is monthly demand variance by the MG’s consumption patterns and $W_i^{\left(t\right)}$ is expected energy production efficiency by the weather of $M{G}_i$ at time step $t$. The profits of energy transactions can be expressed as

$$\eta =B_{sale}+B_{purchase},$$

(23)

$$B_{sale}=B_{trading}+B_{storage},$$

(24)

$$B_{purchase}=B_{margin}^{Maingrid}+B_{margin}^{trading},$$

(25)

where $\eta$ is energy transaction profit of MGs, $B_{sale}$ is benefit of energy sales, $B_{purchase}$ is benefit of energy purchase, $B_{trading}$ is benefit of the energy trading from the seller’s point of view, $B_{storage}$ is benefit of the energy storage for expected profit, $B_{margin}^{Maingrid}$ is benefit of margin compared to trading with the main grid and $B_{margin}^{trading}$ is benefit of the price margin of the transaction. The $\eta$ is used to keep a balance of the profits between supplier and consumer. The high profits can be achieved in the MGs when they conduct a price-efficient transaction with appropriate energy. Then, the benefits from energy trading are given by

$$B_{trading}=p_i^{\left(t\right)}{T}_i^{\left(t\right)},$$

(26)

$$B_{storage}=E_{i,remain}^{\left(t\right)}{\mathrm{B}}_{exp}^{\left(t\right)},$$

(27)

$$B_{margin}^{Maingrid}=\left(p_{maingrid}-p_{i,bid}^{\left(t\right)}\right)T_i^{\left(t\right)},$$

(28)

$$B_{margin}^{trading}=\left(p_{trading}^{\left(t\right)}-p_{i,bid}^{\left(t\right)}\right)T_i^{\left(t\right)},$$

(29)

where $E_{i,remain}^{\left(t\right)}$ is the remaining energy after MG’s energy transaction at time step $t$, $p_{maingrid}$ is energy price of the main grid, $p_{i,bid}^{\left(t\right)}$ is the bid energy price of $M{G}_i$ at time step $t$, and $p_{trading}^{\left(t\right)}$ is energy trading price of $M{G}_i$ at time step $t$. The value of $p_{trading}^{\left(t\right)}$ is chosen in the middle of the values between supplier and consumer price of energy. The DDQN reward can be described as

$$r=\left\{\begin{array}{cc}0\hfill & V_p^{\left(t\right)}<V_p^{thr},V_d^{\left(t\right)}<V_d^{thr}\\ \alpha ;\hfill & V_p^{\left(t\right)}<V_p^{thr},V_d^{\left(t\right)}>V_d^{thr} or V_p^{\left(t\right)}>V_p^{thr},V_d^{\left(t\right)}<V_d^{thr}\\ 1;\hfill & V_p^{\left(t\right)}>V_p^{thr},V_d^{\left(t\right)}>V_d^{thr}\end{array}\right.$$

(30)

where $\alpha$ is a constant value between 0 and 1, $V_p^{\left(t\right)}$ is profit at time step $t$, $V_p^{thr}$ is a threshold of the profit, $V_d^{\left(t\right)}$ is dependency at time step $t$ and $V_d^{thr}$ is a threshold of dependency.

2.3.3. Energy Trading Scheme with the Double Kelly Strategy

The proposed energy trading scheme can be modeled by the double Kelly strategy and the DDQN for the improvement of the energy trading system. The flowchart of the proposed scheme is represented in Figure 3.

The energy trading partners are chosen by the proposed energy trading scheme. The quantity of energy trading is determined by the double Kelly strategy. The proposed DDQN algorithm-based energy trading scheme is summarized in Algorithm 1.

Algorithm 1 Proposed DDQN algorithm-based energy trading scheme

1: Initialize agent parameters, DDQN networks ($Q_{evaluation},Q_{target})$, buffer memory $B$

2: Initialize state parameters $stat{e}_i^{\left(t\right)}=\left[S_i^{\left(t\right)},T_i^{\left(t\right)},P_i^{\left(t\right)},M^{\left(t\right)},W_i^{\left(t\right)}\right]$

3: for $Episodes=1,2,\dots ,N$ do

4:   if $Episodes>Episode{s}_{Max}$ then

5:    Get initial $stat{e}_i^{\left(t\right)}=\left[S_i^{\left(t\right)},T_i^{\left(t\right)},P_i^{\left(t\right)},M^{\left(t\right)},W_i^{\left(t\right)}\right]$

6:    for $Steps t=1,2,\dots ,M$ do

7:      if $Steps<Step{s}_{Max}$ then

8:       Estimate $generatio{n}_i^{\left(t\right)},deman{d}_i^{\left(t\right)}$ from $W_i^{\left(t\right)},M^{\left(t\right)}$

9:       Compute $P_i^{\left(t\right)}$ by (21) and (22)

10:       Compute $K_{i,double}^{\left(t\right)}$ by (1)–(15)

11:       Get $T_i^{\left(t\right)}$ by Equation (16)

12:       Get $stat{e}_i^{\left(t\right)}=\left[S_i^{\left(t\right)},T_i^{\left(t\right)},P_i^{\left(t\right)},M^{\left(t\right)},W_i^{\left(t\right)}\right]$

13:       Send $stat{e}_i^{\left(t\right)}$ to DDQN

14:       Remember memory in $B$

15:       if $B>B_{max}$ then

16:         Replay memory and update target network

17:       end if

18:       Select action $a_i^{\left(t\right)}$ with $\epsilon -greedy$ strategy and send to the environment

19:       Execute energy transactions by $a_i^{\left(t\right)}$

20:       Check $overflow$ and conduct transaction

21:       Compute $V_p^{\left(t\right)} and V_d^{\left(t\right)}$

22:       Compute reward $r^{\left(t\right)}$ by (23)–(30)

23:       Observe next state $stat{e}_i^{\left(t+1\right)}$ and reward $r^{\left(t\right)}$

24:       Store ($stat{e}_i^{\left(t\right)},a_i^{\left(t\right)},r^{\left(t\right)},stat{e}_i^{\left(t+1\right)}$) and done

25:      end if

26:    end for

27:   end if

28: end for

3. Simulation Results

3.1. Settings in Agents, Data, and System Parameters

It is assumed that all agents have the capability of storing and producing energy. The agents are set up with four MGs and consist of two types of traders. The types of traders are composed of consumers and suppliers [31,32,33]. A consumer-type MG represents the MG which consumes energy most of the time while a supplier-type MG denotes the MG which produces a large trading energy quantity.

It is well known that energy demand and production are highly affected by seasonal data settings reflecting weather conditions [31,32,33,34,35,36]. In the simulations, their impacts have been taken into account for the justification of the proposed strategies by obtaining practical results in non-stationary operating environments.

In

Table 2. List of hyperparameters and values.

Parameter	Value	Parameter	Value
Episodes	2500	Discount rate	0.99
Simulation step	365	Epsilon minimum	0.01
Learning rate	1 × 10−3	Epsilon decay	0.999
Optimizer	Adam	Batch size	128
Layers	1D-CNN	Experience replay memory size	30,000

, hyperparameters used in the simulations are presented for the proposed ETS.

3.2. Performance Metrics

In this paper, the three kinds of performance metrics are employed to evaluate the proposed scheme. They include scores according to the DDQN algorithm, energy dependence on the main grid and total profits of MGs over a year. They are expected to have a critical impact on the improvement of profits and reducing main grid dependency in the MG with optimized ETS.

The final rewards, considering the main grid dependency and the profits through energy trading, are represented by the DDQN training scores. The proposed ETS is focused on balanced energy schedules which are considered alongside the profits and independence in the MGs. Therefore, an efficient ETS can be achieved when energy independence and profits from energy trading are balanced. A high score can be achieved by reducing the energy dependency and producing a highly lucrative level of the transaction. Therefore, the high score can bring about the result of activating the energy trading market in the MG.

As the paradigm shifts from centralized to distributed energy control, the ETS should act to the reduce the dependence on the main grid [36]. The off-grid MG has been spotlighted in order to ensure the stability of the entire society’s energy supply system. In addition, independence on the main grid is required for quick restoration in the case of an accident in the power systems.

The total profits in energy transactions are a transparent measure in the ETS. In the energy market, increasing the quantity of energy trading and total profits of MGs have been recommended. The proposed ETS has been devised to pursue the profit of the entire MG rather than a monopoly for the long-term. Therefore, a high profit can be achieved by accurate prediction of the entire energy market and appropriate transactions in the ETS.

3.3. Simulation Results in Energy Trading Scenario

The improvements in independence on the main grid and profits are shown from the simulation results over a year-long period. The simulations were repeatedly operated until a plateau was reached due to the cost of the DDQN methods. In Figure 4, the three strategies (the proposed double Kelly strategy, EFD [15], and CCPV [15]) of the ETS are compared in terms of the DDQN learning score. A high score of convergences is revealed by the DDQN-based energy trading in the proposed strategy. An average score of 260 after episode 350 is shown by the proposed strategy. It is confirmed that a balance between the profits via energy trading and energy independence can be achieved by the proposed system. Similarly to the proposed strategy, an average score of 250 is shown in the CCPV-ETS. However, a low score is shown in the EFD-ETS because it implements energy trading free from competition.

In Figure 5, the energy quantity of each MG received from the main grid is presented. In Figure 5a, it is confirmed that the proposed strategy can reach a significant reduction in the energy quantity from the main grid, which leads to a meaningfully lower dependency on the main grid. Therefore, there is a high possibility of making a self-sufficient system by the proposed strategy. It can be also interpreted that the double Kelly strategy based ETS acts intending to reduce the dependence of the entire MG by taking drastic trading actions in consideration of the future state while EFD and CCPV strategies only focus on the present state without looking at the long-term perspective. In Figure 5b,c, taken as a whole, the insufficient energy of the MG is supplied by the main grid and can be regarded as partially dependent on the main grid.

In Figure 6, the total quantity of dependent energy on the main grid is presented over a year. A lower value of total quantity represents lower dependency on the main grid. The independence performance of the proposed ETS can be improved by 17% compared to EFD-ETS and 7% compared to CCPV-ETS, respectively. Therefore, the aims of energy independence can be achieved by utilizing the proposed ETS scheme.

In Figure 7, total profits were presented over a year for the proposed ETS, EFD-ETS, and CCPV-ETS strategies. It is confirmed that the proposed ETS can achieve higher profits compared with EFD and CCPV strategies. In the proposed ETS, each MG competes with the others in different situations and aims at increasing social welfare by an equilibrium state. However, the EFD and CCPV strategies do not produce high profits because they are not competitive strategies.

4. Conclusions and Discussion

In this paper, the energy trading issue of establishing self-sufficient energy and an efficient MG system was described. The inefficiencies in energy trading systems of microgrids caused by uncertainties of non-stationary environments can be mitigated by the proposed double Kelly strategy. The DDQN algorithm-based ETS with a double Kelly strategy has been proposed in order to improve the profits and energy self-sufficiency in the microgrid’s energy trading systems for the long-term. From the simulation results, it was verified that the proposed strategy can reduce the dependence on the main grid in non-stationary environments. Additionally, in terms of profitability, significant improvement was demonstrated via appropriate pricing and energy trading quantity. In conclusion, it is worth reiterating that the proposed ETS coordination, based on the day-ahead proper energy trading, can provide a promising approach to profitable and energy self-sufficient solutions for the future microgrid. The proposed scheme can affect establishing an efficient energy trading policy for alleviating the problems of energy imbalance.

A few limitations of the proposed scheme may include depreciation of the energy storage system with time and, additionally, the deprecation of the characteristics of applicable distributed energy resources to practical energy trading systems of the microgrid. It is expected that these limitations can be relieved or overcome by the following techniques: efficient management of surplus energy for the energy storage systems, risk management of each energy resource, precise control of demand and supply system and more sophisticated prediction algorithms of energy trading environments.

Future research can be directed towards security of energy information, which could be improved by conducting distributed ETS rather than a centralized energy transaction control. Additionally, another study related to energy storage systems can be suggested for improving battery efficiency. In addition, different social factors and predictive algorithms can be considered for enhancing the performance of trading efficiency. The proposed energy trading scheme can find its application in establishing efficient and effective energy management and transaction systems for microgrid energy networks.