Blockchain-Driven Real-Time Incentive Approach for Energy Management System

Abstract

In the current era, the skyrocketing demand for energy necessitates a powerful mechanism to mitigate the supply–demand gap in intelligent energy infrastructure, i.e., the smart grid. To handle this issue, an intelligent and secure energy management system (EMS) could benefit end-consumers participating in the Demand–Response (DR) program. Therefore, in this paper, we proposed a real-time and secure incentive-based EMS for smart grid, i.e., RI-EMS approach using Reinforcement Learning (RL) and blockchain technology. In the RI-EMS approach, we proposed a novel reward mechanism for better convergence of the RL-based model using a Q-learning approach based on the greedy policy that guides the RL-agent for faster convergence. Then, the proposed RI-EMS approach designed a real-time incentive mechanism to minimize energy consumption in peak hours and reduce end-consumers’ energy bills to provide incentives to the end-consumers. Experimental results show that the proposed RI-EMS approach induces end-consumer participation and increases customer profitabilities compared to existing approaches considering the different performance evaluation metrics such as energy consumption for end-consumers, energy consumption reduction, and total cost comparison to end-consumers. Furthermore, blockchain-based results are simulated and analyzed with the help of deployed smart contracts in a Remix Integrated Development Environment (IDE) with the parameters such as transaction efficiency and data storage cost.

1. Introduction

The proliferation of energy demand necessitates the effective production and distribution of energy in modern grid infrastructure, i.e., a smart grid with an automated control facility for an energy management system (EMS). Energy management (EM) can be realized using three ways, i.e., energy efficiency, strategic load growth, and Demand–Response (DR). Strategic load and energy efficiency involve long-term planning problems and do not consider real-time planning, whereas DR is a mechanism that controls load in real time [1]. There are several ways to implement load control in real time, such as direct load control, time-based techniques, and many more. In direct load control, an electric utility company (EUC) can switch off/on the end-consumer electric appliances and provide incentives to the end-consumer as per the agreement. Furthermore, the total energy consumption associated with the end-consumer in the time-based techniques remains the same. Only the time to consume energy is changed considering the varying price signal forwarded by the EUC to the consumer. It also changes the shape of the energy load curve with the help of minimization of peak-to-average ratio (PAR) and reduces the energy bill of the end-consumer [2,3]. Moreover, DR can be characterized [4] into price-based DR and incentive-based DR, both of which have been comprehensively investigated in smart grid systems [5,6].

The usage of the DR mechanism can be defined with the time of use (TOU), critical peak pricing (CPP), and real-time pricing (RTP) mechanisms. As per the available literature survey, it is found that end-consumers are comfortable with the TOU mechanism, and system complexity is also less. Still, it suffers from a rebound peak problem, which is not the scenario in RTP. In recent decades, different DR strategies have been presented that aim to control residential houses or commercial buildings [7,8]. For example, Sun et al. [9] study the importance of heating, ventilation, and air conditioning, i.e., (HVAC) with distributions and physical parameters. Next, Zhang et al. [10] presented a service pricing-based load balancing approach for residential end-consumers. In [11], a secure and effective real-time scheduling mechanism is proposed for residential DR. Next, Ruzbahani et al. [12] presented an optimal incentive-based DR program for smart homes. Most studies focused on the deterministic rule, abstract model, or mathematical approach that suffers from various issues. For example, optimality cannot be obtained by the deterministic rules in the dynamic energy systems that can cause financial losses. It is heavily dependent on the operator’s skill and suffers from scalability issues (for example, game-theoretic or MILP optimization) due to the involvement of a high number of binary variables.

To tackle the aforementioned issues associated with optimality and scalability, one of the noteworthy solutions is Artificial Intelligence (AI), which has proved its effectiveness toward optimal decision making utilizing Deep Learning (DL) and Reinforcement Learning (RL). Several RL approaches, such as Q-learning, Deep Q-Network, etc., have been incorporated by researchers worldwide to mitigate the decision-making problems [13,14,15]. Then, Zheng et al. [16] focused on the behavioral coupling of end-consumers by incorporating an incentive-based integrated DR approach for multiple energy carriers. In [17], a priority double deep Q-learning approach is presented to improve residential EMS. Most of the existing techniques used in Q-learning are modeled as Markov Decision Processes (MDPs). Still, it has not been exploited fully with real-time incentives and the accessibility of data for all stakeholders such as smart grid, end-consumers, and EUC. Other challenges, such as confidentiality, security, and privacy, must also be considered for efficient and trustable EM. Therefore, blockchain technology is the only solution to handle all these challenges mentioned above.

Blockchain is a secure, immutable, distributed ledger technique (DLT) that contains a chain of data blocks to mitigate security and trust issues such as single-point-of-failure, anonymity, and data manipulation [18]. It has been adopted to monitor EMS securely in the smart grid environment efficiently. For example, the authors of [19] formulated a Stackelberg game approach for achieving optimal energy pricing for efficient energy trading. In [18], blockchain achievability is presented in a smart grid system. Next, Jindal et al. [1] projected a blockchain-based system, i.e., GUARDIAN, to ensure the security of DR. Later, researchers adopted the decentralized blockchain technology in EMS for residential research areas as well [20]. As a result, existing blockchain-based approaches have several limitations, such as data storage cost (relatively high), high energy consumption, low transaction efficiency, and the requirement of high bandwidth to access data in real time [21,22,23].

Table 1. Comparative analysis of various state-of-the-art EMSs with the proposed.

Author	Year	Objective	Pricing Mechanism	Pros	Cons
Zhang et al. [10]	2020	Presented a load dispatch energy storage method for residential area	Iteration algorithm	Reduced operation cost, convergent	Need to consider energy trading for dynamic energy loads, privacy issues
Kumari et al. [11]	2020	Implemented the smart contract to ensure secure energy trading for smart grid	No mechanism	High scalability, reduced storage cost, and low latency	Should focus on optimal pricing, efficiency, and energy consumption
Zheng et al. [16]	2020	Presented a DR model to obtain the incentives for multiple energy carriers	Incentive-based approach	Improved accuracy, reduced dissatisfaction cost	Reduced energy consumption and transaction efficiency is not focused
Mathew et al. [17]	2021	Proposed a DR learning model for an efficient residential EM	DR-based greedy policy	Optimized peak cost and peak load	Need to implement with larger state space for optimal incentive
Li et al. [19]	2018	Discussed a secure energy-trading system for the Industrial Internet of Things using consortium blockchain	Stackelberg game	Optimized price, secure against double-spending and adversary attacks	No discussion on energy consumption reduction and cost
Hupez et al. [24]	2022	Formulated a game-theoretical approach for efficient energy scheduling in residential communities	Non-cooperative game theory	Optimized incentive and fair	No discussion on energy consumption, data storage cost, and transaction efficiency
Bruno et al. [25]	2022	Presented a residential demand response management for optimal load scheduling	Genetic algorithm	Reduced energy cost and electricity bill	Reliability, data storage cost, and energy consumption need to be considered
The proposed approach	2022	Proposed a real-time incentive approach for EMS using blockchain	Q-learning	Optimal price, incentive, high efficiency, and reliability	-

shows the comparative analysis of energy management systems with the proposed approach, which highlights how the proposed approach surpasses the research gaps such as the reliability, data storage cost, and transaction efficiency of related research work with the help of blockchain and interplanetary file system (IPFS)-based framework. Motivated by the above-mentioned gap, this paper proposed the RI-EMS approach: a Real-time Incentive-based Energy Management System using RL (i.e., Q-learning) and blockchain. The proposed RI-EMS approach stores energy data transactions utilizing an off-chain data storage platform, i.e., IPFS, that improves the scalability, data storage cost, reliability, and throughput of the EM.

1.1. Research Contributions

The following are the research contributions of this paper.

• This paper proposes an RI-EMS approach for DR based on Q-learning to prioritize the experience of an agent and for faster convergence of DR using an epsilon greedy policy.
• A novel real-time incentive mechanism is proposed using a smart contract for the end-consumer to motivate them to participate in DR due to the appropriate and optimal incentives obtained for each participant in the EM.
• The proposed RI-EMS approach is evaluated compared to the conventional approaches in terms of consumer participation, energy consumption reduction, transaction efficiency, and data storage cost.

1.2. Organization of the Paper

The rest of the paper is organized as follows. First, Section 2 highlights the system model and problem formulation of the proposed RI-EMS approach, and Section 3 discusses the proposed RI-EMS approach in detail. Next, Section 4 presents the performance evaluation of the RI-EMS approach. Finally, the paper is concluded with future work in Section 5.

2. System Model and Problem Formulation

This section presents the system model and problem formulation of the proposed RI-EMS approach.

2.1. System Model

The proposed RI-EMS approach (as shown in Figure 1) involves the utilization of a smart grid platform to optimize and preserve energy consumption for consumers with the incorporated blockchain network. Now, energy consumption associated with the consumers ${\sigma }_i$ participating in the energy management scheme can be defined based on the different types of energy load ($E_l$) in the particular locality, i.e., residential or commercial. For that, initially, energy consumption data are considered to optimize the incentive for consumers as residential ${\sigma }^r$ or commercial ${\sigma }^c$. Based on the residential or commercial consumers, we can contemplate the energy consumption to optimize it further using the smart grid.

Therefore, if we consider the case of residential consumers, then energy consumption is affected by various energy loads, i.e., thermal (${\theta }_{{\sigma }^r}$), time-shiftable (${\tau }_{{\sigma }^r}$), power-shiftable (${\mu }_{{\sigma }^r}$), and other non-controllable ($N_{{\sigma }^r}^c$) energy loads. Meanwhile, we assume the energy consumption associated with the commercial consumers to be affected by the controllable (${\beta }_{{\sigma }^c}$) and non-controllable (${\delta }_{{\sigma }^c}$) energy loads. Next, we need to reduce the evaluated energy consumption of residential and commercial consumers with the help of a smart grid. Thus, consumers with higher energy consumption can be given an incentive, which will also encourage them to save more energy in real time [18]. It seems difficult for consumers to save energy due to the involvement of high energy consumption. To achieve the optimum energy consumption, we have formulated an incentive mechanism for the residential and commercial consumers by applying a Q-learning approach in which $ϵ$ greedy policy is applied to attain reduced energy consumption and convergence for a better response from consumers. TOU-based EMS is introduced in a smart grid to incentivize consumers based on their energy usage. Furthermore, to ensure the fair incentive mechanism in the scheme, we have employed a blockchain network enabled with IPFS to store the data in a distributed and immutable manner to add data transactions in the blockchain network further (with the help of a smart contract) [26]. However, before storing the energy consumption data in IPFS, data should be legitimate and authenticated by the introduced validation authority (VA). Once data are authorized by VA, they can be made available for storage in IPFS based on smart contract execution. So, real-time data accessibility and storage can be performed over a blockchain network in a real-time incentive-based energy management scheme using the smart grid.

2.2. Problem Formulation

The proposed RI-EMS approach is a real-time incentive-based energy management system in which s number of consumers $\{{\sigma }_1,{\sigma }_2,{\sigma }_{s{}^{\prime }},\dots \dots ,{\sigma }_s\}$ are categorized into residential ${\sigma }^r$ and commercial consumers (${\sigma }^c$) based on their energy consumption. However, to enable the classification of consumers, energy consumption data can be defined based on the consumers, i.e., residential or commercial. Next, energy consumption associated with the residential consumer is determined by considering the energy loads, i.e., {${\theta }_{{\sigma }^r},{\tau }_{{\sigma }^r},{\mu }_{{\sigma }^r},N_{{\sigma }^r}^c$} of various appliances. Similarly, the energy consumption corresponding to the commercial consumers also depends on the types of energy loads of appliances, which is assumed to be controllable ${\gamma }_{{\sigma }^c}$ and non-controllable $N_{{\sigma }^r}^c$. Therefore, we can mention the various energy loads ($E_l^{{\sigma }^r},E_l^{{\sigma }^c}$) which affect the energy consumption of the residential and commercial consumer. The aforementioned association is represented as follows:

$$\begin{array}{c}\hfill E_l^{{\sigma }^r}=\left\{\begin{array}{cc}\theta ,\hfill & \mathrm{if} \mathrm{thermal} \mathrm{load}\hfill \\ \tau ,\hfill & \mathrm{if} \mathrm{time}\text{-}\mathrm{shiftable} \mathrm{load}\hfill \\ \mu ,\hfill & \mathrm{if} \mathrm{power}\text{-}\mathrm{shiftable} \mathrm{load}\hfill \\ N^c,\hfill & \mathrm{if} \mathrm{non}\text{-}\mathrm{controllable} \mathrm{load}\hfill \end{array}\right.\end{array}$$

(1)

$$\begin{array}{c}\hfill E_l^{{\sigma }^c}=\left\{\begin{array}{cc}\beta ,\hfill & \mathrm{if} \mathrm{controllable} \mathrm{load}\hfill \\ \delta ,\hfill & \mathrm{if} \mathrm{non}\text{-}\mathrm{controllable} \mathrm{load}\hfill \end{array}\right.\end{array}$$

(2)

Thus, we have discussed the variables affecting the energy loads of residential and commercial consumers. Then, based on the several energy loads, we can focus on the energy consumption corresponding to the residential and commercial consumers. Firstly, we have to evaluate the energy consumption associated with the consumers ${ϵ}_{{\sigma }_r},{ϵ}_{{\sigma }_c}$ based on the classification, i.e., residential and commercial. Therefore, the energy consumption of residential and commercial consumers can be calculated considering the energy demand $\epsilon$ of various energy loads at a time interval $\xi$, which can be mentioned as follows:

$${\epsilon }_{{\sigma }_r}^{\xi }={\epsilon }_{{\theta }_{{\sigma }_r}}^{\xi }+{\epsilon }_{{\tau }_{{\sigma }_r}}^{\xi }+{\epsilon }_{{\mu }_{{\sigma }_r}}^{\xi }+{\epsilon }_{N_{{\sigma }_r}^c}^{\xi }$$

(3)

$${\epsilon }_{{\sigma }_c}^{\xi }={\epsilon }_{{\beta }_{{\sigma }_c}}^{\xi }+{\epsilon }_{{\delta }_{{\sigma }_c}}^{\xi }$$

(4)

Next, we have to determine the reduction in energy consumption of residential and commercial consumers to achieve the maximum incentive for optimal energy management using the smart grid. However, the incentive obtained by the residential and commercial consumers depends on the reduction in energy consumption. Thus, the reduction in energy consumption of residential consumers can be calculated as follows:

$${\epsilon }_{{\sigma }_r^{Min}}^{\xi }={\left|{\epsilon }_{{\sigma }_r^a}^{\xi }-{\epsilon }_{{\sigma }_r^o}^{\xi }\right|}^2$$

(5)

where ${\epsilon }_a^{\xi }$ and ${\epsilon }_o^{\xi }$ denote the actual and objective energy consumption at a time interval $\xi$. Furthermore, objective energy consumption is decided based on previous energy usage in energy management. The actual energy consumption of residential consumers can be evaluated based on the types of energy loads along with their energy usage, which is mentioned as follows:

$${\epsilon }_{{\sigma }_r^a}^{\xi }=N_{\tau }^{\xi }{\epsilon }_k+N_{\mu }^{\xi }{\epsilon }_l+N_{N^c}^{\xi }{\epsilon }_m-N_s^{\xi }{\epsilon }_n$$

(6)

Similarly, the reduction in energy consumption of commercial consumers can be determined as follows:

$${\epsilon }_{{\sigma }_c^{Min}}^{\xi }={\left|{\epsilon }_{{\sigma }_c^a}^{\xi }-{\epsilon }_{{\sigma }_c^o}^{\xi }\right|}^2$$

(7)

Here, the actual energy consumption of commercial consumers can be calculated based on the types of energy load, i.e., controllable and non-controllable. As we have not considered the shiftable types of energy load for commercial consumers (not included in the scope of this research work), the calculation of actual energy consumption of commercial consumers with the number of controllable and non-controllable energy loads can be represented as follows:

$${\epsilon }_{{\sigma }_c^a}^{\xi }=N_{\beta }^{\xi }{\epsilon }_k+N_{\delta }^{\xi }$$

(8)

Thus, we have considered the N and P number of energy loads for residential and commercial consumers to reduce energy consumption, which further leads to efficient and optimal energy management using a smart grid. In the proposed system, the main criteria of the smart grid are to provide an incentive to the consumers based on the reduction in energy consumption ${\sigma }_r^{Min}$ and ${\sigma }_c^{Min}$. Therefore, we have deduced the objective function $C^O,C^{O{}^{\prime }}$ to optimize the real-time incentive for consumers ${\sigma }_{r,c}$ with the help of a smart grid at a time interval (in hours) $\xi$∈$\{1,2,\dots \dots ,24\}$, which can be mentioned as follows:

$$Optimize(C^O,C^{O{}^{\prime }})=\sum _{\xi =1}^{24}\sum _{(N,P)=1}^{4,2}{\epsilon }_{{\sigma }_{r,c}^{Min},(N,P)}^{\xi }\ast {\rho }^s$$

(9)

where N and P number of energy loads affecting the energy consumption of residential and commercial consumers are considered for optimizing the incentive price based on the obtained reduced energy consumption ${\sigma }_{r,c}^{Min}$ with the help of smart grid ${\rho }^s$. Moreover, the energy consumption data of residential and commercial consumers are stored in IPFS immutable data storage after being made legitimate by VA. After being authenticated by VA, a smart contract runs to check the energy data’s validity that must be stored in the IPFS. As a distributed and secure ledger, the blockchain network stores data transactions with improved cost-efficiency with the help of integrated IPFS protocol. Furthermore, the proposed incentive mechanism ensures the optimal energy management of the consumers with the help of a smart grid. The real-time incentive mechanism works on the principle of an optimal Q-value and action-value function determined to achieve the reward for consumers at a particular state in a dynamic environment. Furthermore, the employed 5G network helps to provide the incentive to consumers with high efficiency, availability, and reliability in the blockchain-based energy management system. A 5G wireless network, along with its low latency, high availability, and high data rate (DR) features, is being considered in the blockchain and IPFS-based energy management scheme.

3. The Proposed Approach

Figure 2 shows the proposed RI-EMS approach, i.e., a blockchain-based real-time incentive energy management scheme, which is divided into a 3-layered architecture consisting of an energy layer, incentive layer, and blockchain layer. These layers are explained in detail to provide an overview of the proposed approach, which is represented as follows:

3.1. Energy Layer

The proposed scheme initiates with the energy layer, which involves collecting energy consumption data of residential and commercial consumers to deduce the maximum incentive for using various energy loads. The RI-EMS utilizes the Q-learning-based RL approach to attain the real-time incentive by extracting the Q-value with the help of the Q-table. Consumers’ energy consumption is affected by the usage of various appliances associated with the energy loads, i.e., thermal, time-shiftable, power-shiftable, and other non-controllable loads for residential consumers and controllable and non-controllable energy loads for commercial consumers. The proposed scheme mainly focuses on optimizing the energy consumption associated with residential and commercial consumers using the smart grid. The energy consumption of consumers, along with their corresponding energy loads, can be represented as follows:

$${ϵ}_{{\sigma }_r}\stackrel{}{\to }\{{\theta }_{{\sigma }^r},{\tau }_{{\sigma }^r},{\mu }_{{\sigma }^r},N_{{\sigma }^r}^c\}$$

(10)

$${\sigma }^c=\{{\beta }_{{\sigma }^c},{\delta }_{{\sigma }^c}\}$$

(11)

Moreover, the energy consumption of residential and commercial consumers fluctuates based on the usage of appliances of various energy loads. Therefore, we have formulated the incentive mechanism for consumers based on the reduction in energy consumption explained in the incentive layer. Figure 3a shows the flowchart for the energy layer that mainly indicates the energy consumption associated with several energy loads, which the RL agent handles.

3.2. Incentive Layer—Reinforcement Learning Approach

The incentive layer serves as a middle layer between the energy and blockchain layer to provide real-time incentives to the residential and commercial consumers based on the optimized dynamic energy consumption ${\epsilon }_{{\sigma }_r^{Min}}^{\xi }$ and ${\epsilon }_{{\sigma }_c^{Min}}^{\xi }$ calculated using actual and objective energy consumption. We have applied the RL approach to obtain the minimized cost for residential and commercial consumers based on dynamic energy consumption. Furthermore, Figure 3b depicts that the RL method comprises multiple agents, i.e., residential and commercial consumers, whose main aim is to choose an action that yields the minimized cost in a dynamic environment. To implement the RL method, we need to consider three elements, i.e., state, action, and cost.

Assume S denotes the state set which represents the state of the RL agent, i.e., residential and commercial consumers ($s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi }$) at a time interval $\xi$. Action is defined by A, which signifies the action of consumers ($a_{{\sigma }_r,\xi },a_{{\sigma }_c,\xi }$)∈(${\epsilon }_{{\sigma }_r^{Min}}^{\xi },{\epsilon }_{{\sigma }_c^{Min}}^{\xi }$) to the dynamic environment to obtain the maximized cost. For example, how residential and commercial consumers act in a dynamic environment based on the dynamic energy consumption can decide their optimized cost $C_{\xi }^O$ and $C_{\xi }^{O{}^{\prime }}$. After obtaining the optimized cost, the dynamic environment can be forwarded to the next state ($s_{{\sigma }_r,\xi +1},s_{{\sigma }_c,\xi +1}$). However, we have already discussed the associated energy loads of residential and commercial consumers and how they influence energy consumption. We have evaluated the reduced energy consumption of the consumers with the help of actual and objective energy consumption. Based on the calculated reduced energy consumption, we have deduced an objective function specifying the cost for residential and commercial consumers varying with the reduced energy consumption, and the smart grid ensures the low pricing for consumers.

Therefore, firstly, we can define the optimal policy $\Delta$ for agents, i.e., residential and commercial consumers, to optimize the objective cost evaluated using reduced energy consumption. Thus, the action-value function $Q_{{\Delta }_{{\sigma }_r}}$ for residential consumers considering the state, action, and policy can be represented as follows:

$$Q_{\Delta }(s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })=\sum _{j=\xi +1}^T{\omega }^{j-\xi -1}{C}_{j-1}^O|s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi }$$

(12)

$$\forall s_{{\sigma }_r,\xi }\in S,a_{{\sigma }_r,\xi }\in A$$

(13)

where $\omega$ denotes the discount factor associated with the residential consumers. Similarly, we can calculate the action-value function $Q_{{\Delta }_{{\sigma }_c}}$ for commercial consumers with the help of objective cost $C^{O^{\prime }}$. Moreover, the optimality of the action value for residential and commercial consumers is represented by $Q_{\Delta }^{\ast }(s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })$ and $Q_{\Delta {}^{\prime }}^{\ast }(s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })$.

Furthermore, agents, i.e., ($s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi }$) should take action ($a_{{\sigma }_r,\xi },a_{{\sigma }_c,\xi }$) to maximize the reward or incentive $\eta (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })$ using the Q-learning approach based on the policy at a particular state ($s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi }$). The Q-learning approach works on the principle of Q-value $\Omega (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })$ by preparing the Q-table containing the action ($a_{{\sigma }_r,\xi },a_{{\sigma }_c,\xi }$) and state ($s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi }$). As a result, the flow of the incentive layer with Q-value is considered an important aspect to obtain the optimal price, which is further forwarded to the consumers based on the reduced energy consumption calculated using actual and objective energy consumption. Therefore, the calculation of the Q-value for residential consumers is represented as follows:

$$\begin{array}{c}\Omega (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })\stackrel{}{\leftarrow }\Omega (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })+\beta ({\eta }_{\xi +1}(s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })\hfill \\ \hfill +\omega max\Omega (s_{{\sigma }_r,\xi +1},a_{{\sigma }_r,\xi })-\Omega (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi }))\end{array}$$

(14)

where $\beta$ represents the learning rate which lies in the range of [0, 1] and $\omega$ is considered as the discount factor associated with the action–value pair calculated to maximize the cost objective function of residential consumers. Similarly, we can calculate the optimization of incentive or reward $\eta {}^{\prime }(s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })$ for commercial consumers based on the Q-value, which is expressed as follows:

$$\begin{array}{c}\Omega (s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })\stackrel{}{\leftarrow }\Omega (s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })+\beta (\eta {{}^{\prime }}_{\xi +1}(s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })\hfill \\ \hfill +\kappa max\Omega (s_{{\sigma }_c,\xi +1},a_{{\sigma }_c,\xi })-\Omega (s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi }))\end{array}$$

(15)

Furthermore, Algorithm 1 shows how the Q-learning approach can be used to determine the optimization of the Q-value for residential and commercial consumers with the help of optimal policy in terms of time complexity of O(e) (which represents the number of episodes to compute the optimization of Q-value), which is expressed as follows:

Therefore, we have applied the Q-learning approach to maximize the incentive $\eta (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })$ and $\eta {}^{\prime }(s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })$ for residential and commercial consumers based on the dynamic energy consumption that is considered as the action (${\epsilon }_{{\sigma }_r^{Min}}^{\xi },{\epsilon }_{{\sigma }_c^{Min}}^{\xi }$) taken by the consumers for each state of $s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi }$ at a time interval $\xi$. After obtaining the incentive mechanism for consumers using the Q-learning approach, the secure storage of reduced energy consumption has been explained in the blockchain layer, which focuses on real-time incentive energy storage with the help of the introduced IPFS.

$$\{\delta ,\delta {}^{\prime }\}=argmax(\Omega \left(s_{{\sigma }_r}\right),\Omega \left(s_{{\sigma }_c}\right),\xi )$$

(16)

Algorithm 1 Incentive for Consumers using Q-learning

Input: $s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi },a_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi },Q_{\Delta },Q_{\Delta {}^{\prime }},\xi$ Output: Optimized incentive 1: procedureIncentive_Consumer($s_{{\sigma }_r,\xi },s_{{\sigma }_c,\xi },\xi$ ) 2:     if $\sigma \in {\sigma }_r$ then 3:         for $\xi$ time interval $<0$ do 4:            $AssignQ-value\stackrel{}{\to }0$ 5:            for E dopisode e 6:                Calculate action value for residential consumer 7:                Assign State $s_{{\sigma }_r,\xi }$ 8:                $Q_{\Delta }={\sum }_{j=\xi +1}^T{\omega }^{j-\xi -1}{C}_{j-1}^O|s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi }$ 9:                Compute incentive $\eta (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi })$ 10:                Transit to new state $s_{{\sigma }_r,\xi +1}$ 11:                Compute optimization of Q-value 12:                $\delta =argmax(\Omega (s_{{\sigma }_r,\xi },a_{{\sigma }_r,\xi }))$ 13:            end for 14:         end for 15:     else 16:         for $\xi$ time interval $<0$ do 17:            $AssignQ-value\stackrel{}{\to }0$ 18:            for Episode e do 19:                Calculate action value for commercial consumer 20:                Assign State $s_{{\sigma }_c,\xi }$ 21:                $Q_{\Delta {}^{\prime }}={\sum }_{j=\xi +1}^T{\omega }^{j-\xi -1}{C}_{j-1}^{O{}^{\prime }}|s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi }$ 22:                Compute incentive $\kappa (s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi })$ 23:                Transit to new state $s_{{\sigma }_c,\xi +1}$ 24:                Compute optimization of Q-value 25:                $\delta {}^{\prime }=argmax(\Omega (s_{{\sigma }_c,\xi },a_{{\sigma }_c,\xi }))$ 26:            end for 27:         end for 28:     end if 29: end procedure

3.3. Blockchain Layer

Ethereum blockchain, as a secure and decentralized platform, is introduced to ensure secure and real-time incentive energy management for consumers implemented with a value-based Q-learning algorithm. To accomplish real-time data accessibility and energy consumption data stored securely in the blockchain, IPFS as an immutable peer-to-peer protocol is employed in the system to improve the scalability and reliability of the communication between multiple agents in the dynamic environment [29]. Initially, VA as an authorizing entity is considered to confirm the identity of consumers participating in energy management. To legitimize the authorization of data storage in IPFS, an intelligent contract run as a self-executable code to check the authenticity of energy consumption data. If it becomes authenticated for data storage, then IPFS as a cost-efficient protocol allocates hash keys ${\varphi }_{{\sigma }_r}$ and ${\psi }_{{\sigma }_c}$ to residential and commercial consumers. Next, the consumers containing the hash keys ${\varphi }_{{\sigma }_r}$ and ${\psi }_{{\sigma }_c}$ provided by the IPFS can use them as data access and storage keys to perform the transactions of real-time energy management in the blockchain network.

Algorithm 2 depicts how energy data can be stored securely with the help of a blockchain network considering the time complexity, i.e., O(s), and O($s{}^{\prime }$) associated with the number of residential and commercial consumers request for data storage. Furthermore, the security of energy management transactions of the consumers needs to be ensured in the blockchain network. For that, we have utilized the pair of keys, i.e., public key and private key of consumers $P{c}_{k_{{\sigma }_r}},P{e}_{k_{{\sigma }_r}}$ and $P{c}_{k_{{\sigma }_c}},P{e}_{k_{{\sigma }_c}}$ using asymmetric public key cryptography to preserve the energy management of consumers in the dynamic environment, which is denoted by $D_E$:

Algorithm 2 Blockchain-based algorithm for secure energy data storage

Input: ${\sigma }^r,{\sigma }^c,IPF{S}^{hk},VA$ Output: Add energy data transactions to the blockchain 1: procedureEnergy_data(${\varphi }_{{\sigma }_r},{\varphi }_{{\sigma }_c},{\sigma }^r,{\sigma }^c$ ) 2:     if $\sigma \in {\sigma }^r$ then 3:         for $x=1,2,\dots ,s$ do 4:            $IPF{S}^{hk}\leftarrow$$data\_requests$(${\sigma }^r$) 5:            ${\sigma }^r\stackrel{authorize}{\leftarrow }VA$ 6:            Execute smart contract 7:            if ${\sigma }^r$∈ authorized then 8:                ${\sigma }^r\stackrel{{\varphi }_{{\sigma }_r}}{\leftarrow }IPF{S}^{hk}$ 9:                $blockchain\stackrel{}{\leftarrow }$$Add\_data$ (${\sigma }^r$) 10:                Secure data storage in the blockchain 11:            else 12:                Invalid consumer 13:            end if 14:         end for 15:     else if $\sigma \in {\sigma }^c$ then 16:         for $y=1,2,\dots ,s{}^{\prime }$ do 17:            $IPF{S}^{hk}\leftarrow$$data\_requests$(${\sigma }^c$) 18:            ${\sigma }^c\stackrel{authorize}{\leftarrow }VA$ 19:            Execute smart contract 20:            if ${\sigma }^c$∈ authorized then 21:                ${\sigma }^r\stackrel{{\psi }_{{\sigma }_c}}{\leftarrow }IPF{S}^{hk}$ 22:                $blockchain\stackrel{}{\leftarrow }$$Add\_data$ (${\sigma }^r$) 23:                Secure data storage in the blockchain 24:            else 25:                Invalid consumer 26:            end if 27:         end for 28:     end if 29: end procedure

$$D^h(({\sigma }_r,{\sigma }_c),D_E)=(({\varphi }_{{\sigma }_r},{\psi }_{{\sigma }_c}),D_E)$$

(17)

$${\phi }^{P{c}_k({\sigma }_r,{\sigma }_c)}(D{s}_d^{P{e}_k^{{\alpha }_r}})(D^h(({\sigma }_r,{\sigma }_c),D_E))=D^h(({\sigma }_r,{\sigma }_c),D_E)$$

(18)

where $D^h$ signifies the hash digest of the energy transactions of consumers ${\sigma }_r$ and ${\sigma }_c$ in the dynamic environment $D_E$. $D{s}_d$ represents the digital signature of consumers associated with their private key {$P{e}_{k_{{\sigma }_r}},P{e}_{k_{{\sigma }_c}}$}. Furthermore, Figure 4 shows the basic working of the blockchain layer in which energy data optimized from the incentive layer are stored securely in the blockchain network through the IPFS intermediary protocol. Then, the smart grid operator manages the energy data that can be forwarded to consumers based on their reduced energy consumption.

4. Performance Evaluation

This section gives an overview of the performance evaluation of the proposed RI-EMS approach. The proposed RI-EMS approach is implemented with python high-level programming language on the Windows operating system with the configuration of Intel(R) Core(TM) CPU of 2.60 GHz and 8 GB RAM to maximize the incentive for consumers based on the Q-learning approach considering the performance metrics such as energy consumption for end-consumer, energy consumption reduction, and total cost comparison. Furthermore, blockchain-based results are evaluated and analyzed by deploying the smart contracts in Remix IDE with the help of parameters such as transaction efficiency and data storage cost.

4.1. Dataset Description

The performance evaluation of the proposed RI-EMS approach is conducted using the standard dataset, i.e., Open Energy Information (openEI) [30]. It contains energy consumption data for residential houses and commercial buildings as well. Then, the pre-processing of the energy data is performed with a sci-kit-learn library to tackle noise, Not-a-Number (NaN), missing values, duplicate values, etc. Next, the critical load data (such as AC and other appliances) is obtained from Pecanstreet [31]. Then, hourly energy prices are considered from PJM Data Miner as 2nd August 2022 [32]. Finally,

Table 2. Simulation settings.

Particular	Values
$\xi$	1 h
Peak hour	5 PM to 12 PM
Mid-peak	8 AM to 5 PM
Off-peak	12 AM to 8 AM
${\delta }_{\mathfrak{C}}$	0.01
$\varphi$	0.001
$\beta$	{0,1}

shows the several simulation parameters considered for implementing and predicting the results for the proposed RI-EMS approach.

4.2. Energy Consumption Reduction and Comparative Analysis

Figure 5a highlights the energy consumption of the end-consumers by considering the distinguished non-controllable and controllable energy loads. In the proposed approach, the energy consumption of commercial consumers has been calculated with the help of controllable and non-controllable energy demand. Furthermore, the reduction in energy consumption is determined using the respective consumer’s actual and objective energy consumption. The graph depicts the acquired energy consumption of the proposed approach in a time interval (hours) of [0, 25]. It can be observed from the graph that a controllable energy load yields higher energy consumption than non-controllable energy consumption, which leads to the increased incentive of consumers in the case of a non-controllable energy load.

Figure 5b presents the energy consumption reduction due to the incentive mechanism used in the proposed approach. Here, energy demand is marked in orange color, and consumption of energy by the consumer is marked in green. The dotted line represents the hourly energy prices. This graph depicts the consumption reduction in peak hours and the increase in consumption in non-peak hours; for example, in the morning (1 AM to 8 AM), consumption is high, and during peak hours, consumption is reduced to receive more incentives from the consumer. Furthermore, the proposed RI-EMS approach is compared with the baseline approach such as Gurobi optimizer [33] by having the same simulation parameters setting. Figure 6 depicts the costs comparison, which comprises total energy consumption reduction and discomfort costs to the end-consumer. It is evident from the graph that the proposed RI-EMS approach learns through a trial and error mechanism and performs well with the increasing number of episodes compared to the baseline model.

4.3. Transaction Efficiency

Figure 7a shows the transaction efficiency comparison considering two scenarios in which one scenario is to perform data transactions in the proposed RI-EMS approach with IPFS. Another scenario focuses on performing the data transactions in the proposed approach with blockchain. It can be perceived from the scenarios that transaction efficiency seems to lie at the same level when fewer data transactions are performed between multiple agents. However, with the exponential increase in the number of data transactions, the proposed RI-EMS approach with IPFS exhibits quite improved transactions efficiency compared with the proposed scheme with blockchain. This is because IPFS works on generating a hash, which must be assigned to the consumers for secure and cost-efficient data storage.

4.4. Data Storage Cost

In this subsection, we have focused on the data storage cost of the RI-EMS approach to ensure cost-efficient energy management for consumers. Therefore, we have focused on the data storage cost of the Ethereum blockchain network, which is a decentralized and secure platform. Initially, we highlight an important metric, i.e., gas price for a single word, which is denoted by $G{p}_w$. Furthermore, the gas price $G{p}^K$ for 1 KB of energy consumption data correlates with $G{p}^w$ in which $G{p}^K$ can be calculated as $20\ast {10}^{3}Gas$ and $G{p}^w$ can be written in the form of expression $(2^{10}/256)\ast (20\ast {10}^3)Gas$. Furthermore, data storage cost $C^W$ associated with W number of words in a blockchain can be computed with the parameters gas price and Ethereum price ($g{s}_{bc},E{T}_{bc}$). Therefore, considering ether value (Ev) as ${10}^9$, data storage cost can be expressed in the form $(w\ast G)/Ev$ to calculate the cost in USD as $(g{s}_{bc}\ast C^W)\ast E{T}_{bc}$ [34].

Storage Cost Analysis

The aforementioned computation for data storage cost proves that using blockchain as a data storage platform incurs huge costs, which can demotivate consumers from utilizing the energy of appliances associated with various energy loads. As a result, Figure 7b shows the data storage cost analysis of the proposed RI-EMS approach considering the data storage as blockchain and IPFS. Finally, the graph exhibits relatively low storage cost when using IPFS as data storage with the proposed RI-EMS approach. Moreover, when fewer consumers are involved in the energy data transactions, then the data storage cost for both platforms lies at the exact alignment. Still, with the exponential surge in the number of energy data transactions, the requirement of data storage cost for the proposed RI-EMS approach with IPFS is relatively lower than the blockchain data storage. The main reason for the cost-efficient behavior of IPFS is that it stores consumers’ energy data by generating a hash, which requires a lower cost than the blockchain (stores a whole block of data).

5. Conclusions

The growth of smart homes has increased the research on EMS across the globe. So, in this paper, we presented an incentive-based EMS for smart grid, RI-EMS integrated with blockchain technology in real time. We have adopted the DR-based Q-learning approach to optimize the incentive for residential and commercial consumers based on the calculated reduction in energy consumption. We have categorized consumers based on the several energy loads to obtain insights into energy consumption. Moreover, we have formulated a real-time incentive mechanism based on the action-value function and Q-value applied using the Q-learning approach implemented in the python programming language to obtain the reward or incentive for consumers. The consumer incentive mechanism has been optimized based on the $ϵ$ greedy policy to guide multiple agents for better convergence. Finally, the performance of the proposed RI-EMS approach is simulated against important metrics, i.e., consumer participation, energy consumption reduction, and total cost comparison to end-consumers. Next, blockchain-based results are implemented by deploying the smart contracts in Remix IDE in terms of transaction efficiency and data storage cost.

In the future, we will implement a DL model with the Q-learning approach to obtain the optimum energy consumption for consumers managed by the multiple agents. DL and Q-learning approaches can improve the incentive for consumers monitored by multiple agents. Furthermore, we can consider a real-time and dynamic scenario to implement the blockchain-based technology for efficient and optimal EM in smart homes.