Hierarchical Control for Microgrids: A Survey on Classical and Machine Learning-Based Methods

Abstract

Microgrids create conditions for efficient use of integrated energy systems containing renewable energy sources. One of the major challenges in the control and operation of microgrids is managing the fluctuating renewable energy generation, as well as sudden load changes that can affect system frequency and voltage stability. To solve the above problems, hierarchical control techniques have received wide attention. At present, although some progress has been made in hierarchical control systems using classical control, machine learning-based approaches have shown promising features and performance in the control and operation management of microgrids. This paper reviews not only the application of classical control in hierarchical control systems in the last five years of references, but also the application of machine learning techniques. The survey also provides a comprehensive description of the use of different machine learning algorithms at different control levels, with a comparative analysis for their control methods, advantages and disadvantages, and implementation methods from multiple perspectives. The paper also presents the structure of primary and secondary control applications utilizing machine learning technology. In conclusion, it is highlighted that machine learning in microgrid hierarchical control can enhance control accuracy and address system optimization concerns. However, challenges, such as computational intensity, the need for stability analysis, and experimental validation, remain to be addressed.

1. Introduction

Distributed generators (DG) can effectively improve the utilization efficiency of clean energy, accelerate the energy transformation to be more sustainable, and reduce generation costs. Strategically placing distributed generators (DGs) within the power systems can yield several benefits, such as reducing peak operating costs and power losses, improving voltage distribution, meeting load requirements, and enhancing overall system reliability and integrity [1,2]. However, the deployment of DGs also faces challenges. DGs may bring adverse effects such as inrush current, voltage deviation, and voltage fluctuation in the distribution networks. For example, wind and solar energy are inherently random and intermittent, making it difficult to provide sustained and stable power. The instability of energy output may weaken the role of DGs [3,4]. Therefore, once these distributed systems are not effectively controlled, the stability of the system can be significantly affected [5].

To address such challenges, scholars put forward the concept of a microgrid [6]. A microgrid is a small power generation system composed of distributed power sources, energy storage devices capable of bidirectional transmission, efficient energy conversion equipment, associated loads, and monitoring and protection equipment for the operation [7]. Microgrids can successfully solve problems caused by multiple DG units, making the large-scale application of DG systems possible [8]. A microgrid system can be operated in islanded mode or grid-connected mode. In the islanded mode, all reactive and active power required for loads connected to the microgrid are supplied by the DGs [9], and proper distribution of power needs to be in accordance with the capacity of DGs. In grid-connected mode, the microgrid gets the system frequency and follows connection requirements as specified in the grid codes [10]. For islanded microgrids, it is very critical to maintain the stability of the system and realize load power sharing among multiple parallel DG units [11]. Generally, more control is required than in grid-connected microgrids. If the power cannot be accurately shared or voltage deviation exists, system stability and power quality will deteriorate. In addition, the voltage and frequency of the islanded microgrid are no longer controlled by the main grid [12]. The power output fluctuations of intermittent DGs can lead to severe deviations in frequency and voltage without proper control strategies. In addition, economy, power quality, and power flow need to be considered. Therefore, several hierarchical control strategies have been proposed for different operating conditions, which mainly include primary, secondary, and tertiary control [13].

Artificial intelligence (AI) and big data applications have been widely used in recent years to improve traditional control systems. AI uses machine learning (ML) to process large amounts of data input, becoming one of the most popular solutions [14,15]. ML technology makes decisions based on measured data and can achieve more efficient and safe control under complex conditions [16,17]. Therefore, ML technology with powerful data processing and computing power have the potential to realize the full utilization of multiple DG systems [18]. For the hierarchical control system of a microgrid, ML technology has broad application prospects [19]. For example, in the primary and secondary levels of control, ML algorithms can be used to improve the accuracy of control parameters in control loops to achieve optimal control. In the tertiary control, a ML algorithm can be used to generate an optimal reference point for achieving improved operation in terms of economy, efficiency, and reliability among many others.

This paper aims to provide a comprehensive analysis of recent research on microgrid hierarchical control, specifically focusing on the control schemes and the application of machine learning (ML) techniques. Existing literature includes some works summarizing the application of AI techniques in microgrids. For instance, ref. [7] focused more on artificial neural networks (ANN) in microgrid ML techniques, while [16] reviewed ML application for network attack protection only. Additionally, ref. [20] discussed the usage of AI techniques in microgrids, with a particular emphasis on energy management and load forecasting. However, there is currently no literature that thoroughly summarizes and compares the various aspects of ML techniques in microgrid hierarchical control. Therefore, this paper tries to establish a generic structure based on ML techniques for researchers in primary and secondary control sections to make it easier for researchers to understand the control logic of ML techniques in hierarchical control. Another purpose of this article is to compare and analyze the advantages and disadvantages, control structure, and validation environment of ML technology applied at each level of hierarchical control, to make it easier for readers to clearly comprehend, and to analyze the future technology trends. The focus of this paper is to review the application of ML and other control methods in different control levels of microgrids over the past years. The main contributions are as follows.

• Summarizing the main control modes in the hierarchical control of microgrids through a literature survey over the past five years;
• Analyzing the application status of ML technology in each level of control and reviewing the application of different ML technologies;
• Reviewing, analyzing, and discussing the communication problems of the hierarchical control system.

2. Overview of Microgrid Control

Hierarchical Microgrid Structure

Figure 1 shows the principle of microgrid hierarchical control, which can operate islanded as well as grid-connected, and combined heat power (CHP), photovoltaic system (PV), wind power system, and energy storage system (ESS), etc., and can be used as the basic unit of a microgrid power generation system. IEEE 1547 [21] provides a reference standard for building microgrids and interconnecting them to the grid. G99 [22] is the UK standard for grid integration of DG. IEEE 2030.7 [23] provides operational test verification of DG control systems. IEC 61850 [24] provides a reference standard for communication requirements with data models [25]. IEC 62351 [26] develops security protocols for microgrid communication systems. The hierarchical control in the Figure 1 consists of primary, secondary, and tertiary control. Primary control maintains a stable voltage/frequency and does not require a communication link because it operates in a local control structure of a DG, also known as local control (LC) [27]. Droop control is widely used in local control. As for conventional droop control, due to the influence of DG feeder line impedance mismatch, the problem of active and reactive power sharing is unavoidable. Voltage frequency deviations generated by primary control are compensated by secondary control [28]. Centralized control and distributed control are two types of secondary control. The central controller is a clear sign of centralized control, which can be used to make optimal control decisions for the operation of the microgrid [9]. In distributed secondary control, all DGs use local information from adjacent DGs to keep the voltage/frequency at rated values. The tertiary control operates at the highest level in the control hierarchy aiming to improve, for example, the power quality by monitoring the energy exchange between microgrid and the main grid, ensuring safe use and economic benefits of the users [14]. No matter which scheme the microgrid operates in, analyzing the above problems is indispensable. Therefore, it is essential to improve the control of the microgrid.

In Figure 2, the different control modes applied to each control layer are summarized. The control mode based on ML technology has become a leading research direction of scholars in past years, especially at the secondary and tertiary control level.

3. Review of Hierarchical Control Methods

3.1. Primary Control

Primary control is the lowest control level of hierarchical control that achieves stable microgrid operation by maintaining the power sharing of DGs. Droop control is widely used in primary control, but its accuracy and dynamic performance have to be improved. In the field of power systems, the ML algorithm has been mainly applied in the areas of load/power prediction [29,30], grid fault detection, and diagnosis, while its role in fields such as operation control remains still to be explored. The application of ML algorithms is discussed extensively in the following sections.

3.1.1. Traditional Primary Control

Due to the influence of power lines impedance mismatch, the problem of active and reactive power sharing is unavoidable using conventional droop control [31]. In [32], the authors used the measured current instead of active power and reactive power in droop control in order to distribute the reactive power. However, the frequency deviation is not considered. A VI (voltage current)-based droop control based on global positioning system (GPS) timing has been proposed in [33], where the voltage and current measurements are used instead of active and reactive power used in traditional droop control. Although a precise distribution of reactive power can be achieved, this method ignores the impact of unbalanced loads on the system and is not suitable for complex microgrid systems. In [34], the authors proposed a dispatchable droop control method, which can realize automatic power regulation in a short time on DG so that the system can intervene the control on a large time scale. In [35], it was proposed that the virtual multi-slack (VMS) droop control, which used one physical slack generator directly to control the magnitude and phase angle of its bus voltage, can significantly improve voltage and frequency stability and achieve accurate distribution of the reactive power. However, this method is susceptible to interference from external factors, such as virtual slack matrix. In [36], droop control based on f-P/Q was used to extend the scope of application under resistive–inductive and resistive–capacitive coupling in the microgrid. In [37], a distributed mixed voltage angle and frequency droop control was proposed to achieve an accurate reactive power distribution. However, the system is prone to measurement losses. In [38], an adaptive droop control was proposed, which sets the current and voltage controller in traditional droop control to achieve an accurate power distribution. However, this undoubtedly increases the cost of the control process. In [39], a model predictive control was used to obtain droop control gain. The droop control gain is no longer a constant value, but it is dynamic, and a more accurate reactive power distribution can be achieved. However, this method requires setting of an additional control center, which increases the control costs. In [40], a band-pass filter-based droop control was proposed where the method can enhance voltage and frequency regulation by reducing response time, overshoot, and steady-state error. A multivariable-droop control was proposed in [41], which can achieve low steady-state error and a fast response by decoupling of d- and q-axis currents.

In [42], the concept of an adaptive virtual impedance was proposed; adding virtual impedance to droop control is one of the common improvements. The virtual impedance scheme is achieved by adding a feedback control structure to the inverter control, which is equivalent to a series of analogy resistors or reactance in the line, but the actual line parameters remain unchanged. Hence, the utilization of virtual impedance is a preferable alternative to incorporating transformers or large inductors as it eliminates the introduction of additional power losses. However, since the line parameters are unknown, the virtual impedance values are difficult to determine and in practice are usually chosen to be much larger than the line parameters, which increases the equivalent output impedance of the inverter, resulting in a significant drop in system voltage, having a negative impact on the stable operation of the microgrid. How to select the virtual impedance accurately has become one of the hot topics in research. In [43], they used an adaptive virtual impedance based on a consensus algorithm, which can achieve reactive power distribution without measuring the line impedance information at any time. A two-level adaptive virtual impedance on a GPS synchronization-based controller was used in [44], where the system has a better stability, but it has large communication burden. The adaptive virtual impedance method can improve the reactive power allocation problem and improve the stability of the system, but an accurate adaptive coefficient should be obtained.

3.1.2. Application of ML in Primary Control

As a further improvement of the distributed control, AI-based ML schemes use reward feedback to assess the quality of solutions [45]. AI offers plenty of opportunities to enhance the hierarchical control in islanded microgrids. In [46], the authors proposed a data-driven primary control-based scheme that transforms the control process into a convex optimization problem. This scheme can improve transient performance, while providing power sharing and voltage and frequency restoration where the parameters of the feeder line are not required. The authors in [47,48] proposed an ANN-based droop control for optimal droop gain and improved the distribution accuracy, whereas [47] used a feedforward neural network (FFN) under AC microgrid and [48] used an ANN under DC microgrid conditions. In [49], the authors added an ANN-based controller in the current loop, which can realize better DC bus voltage regulation. In [50], the authors proposed a discrete time distributed neural network (NN); based on a data-driven application in the primary control, a new control structure is adopted, which uses neural network to learn the control method rather than just using traditional static droop control. Authors in [51] proposed an adaptive compensation control strategy based on the adaptive neuro-fuzzy inference system (ANFIS), to get better fuzzy rules, which is demonstrated to improve the system stability. In [52], a novel distributed reinforcement learning (DRL) strategy was proposed to coordinate the current sharing and voltage restoration. In [53], the authors proposed an intelligent weighted power dispatching control method assisted by ML to control the active power dispatch of a diesel generator, battery storage system, and largescale photovoltaic system, which ensures the power balance of the system. A dynamic droop control strategy based on distributed data-driven Q-learning technology was proposed in [54], which can realize an independent compensation of voltage frequency. A summary of ML-based techniques for primary control in MGs is given in

Table 1. Summary of ML-based techniques for primary control.

Ref.	Method	Structure	Advantages	Disadvantages	Demonstration
[46]	Convex optimization method based on data-driven control	Distributed	Improve transient performance, without physical parameters of grid	No plug-and-play, poor scalability	Real-time experiment
[47]	Droop control based on feed forward neural network	Centralized	Improve the accuracy of power-sharing	No stability analysis	Simulation
[48]	Droop control based on artificial neural network	Centralized	Voltage stability can be maintained independently while power sharing is achieved	No stability analysis, ignore communication delay	Simulation
[49]	ANN-based controller in current loop	Distributed	Improve power-sharing between battery and supercapacitor, maintain state-of-charge within boundaries	No stability analysis	Real-time experiment
[50]	Discrete-time distributed NN based data-driven control	Distributed	Achieves an optimal trade-off between voltage regulation and reactive power sharing	The existence and convergence analysis of integrated optimization and dynamic consensus parts are neglected	Simulation
[51]	An adaptive compensation control strategy based on ANFIS	Distributed	Improve the system stability and robustness	Ignore communication delay	Real-time experiment
[52]	A DRL-based control on current sharing and effective voltage restoration	Distributed	High computational efficiency and robustness, the best solution can be found	The effect of line impedance is ignored	Simulation
[53]	Intelligent weighted power scheduling control aided by ML	Distributed	A new control structure with smaller active power deviation	Large amount of calculation	Simulation
[54]	Data-driven Q-Learning	Distributed	Achieve autonomous frequency synchronization and voltage restoration	Ignore communication delay	Simulation

. Figure 3 also illustrates the primary control of a microgrid based on ML, where the primary control agent sends voltage and frequency signals to the voltage and current control circuit. The system will operate according to the signals above by outputting active and reactive power values. At this point, the agent will use the P and Q values collected by the system as state values to determine whether they meet the control requirements, and if they do not meet the requirements, the output u, f will be adjusted, i.e., the agent will update the output voltage and frequency based on the parameters received until the best strategy is obtained. In Figure 3, the main ML algorithms used can be classified as neural networks, reinforcement learning, supervised learning, unsupervised learning, and semi-supervised learning. Previous studies [17,19,20,55] have provided insights into the methods of ML and the underlying principles of different algorithms. The agent can be based on droop control outputs such as [47,48,51,53,54]. The data-driven based control schemes are also a promising alternative to the traditional droop control strategies such as [46,50].

3.2. Secondary Control

The secondary control has the role of detecting, protecting, and effectively restoring the operating system in order to achieve a stable operation with a main function of voltage and frequency compensation for the primary control [4,5]. Studies on traditional controllers, such as proportional integral controller (PI), model predictive control (MPC), and consensus, have rarely stopped proposing improvements, and at the same time, ML-based secondary control has been proposed and emerged.

3.2.1. Traditional Secondary Control

In [56], the authors used a proportional integral controller (PID) to compensate for the voltage and frequency and the authors in [57] proposed to use a frequency compensation method based on PI-particle swarm optimization (PSO) control, which collected the frequency deviation and sent it to the PI-PSO controller afterwards in order to get the compensation value. The PSO algorithm also realized the optimal values of PID control parameters, which reduced the settling time, overshoot, and oscillations. In [58], the paper proposed a PI-consensus controller to minimize the error of the voltage and reactive power. In [59], the authors proposed a cooperation control based on a PI controller to restore the voltage and frequency to the rated value without any primary-level droop control. In order to improve the traditional PID-based control scheme, since it requires manually rectified parameters, ref. [60] proposed the application of fractional-order integral proportional derivative with filter (IPDF) based on the imperialistic competitive algorithm (ICA); ref. [61] proposed a two-stage fractional-order PID controller based on imperialistic competitive algorithm; ref. [62] proposed a control strategy using fuzzy tilted integral derivative (FTID) and filter plus double integral (FTIDF-II); and ref. [63] proposed a control scheme based on tilted differential filter–tilted integral differential filter (TDF–TIDF) controller.

In [64], the authors used an MPC on primary and secondary control, which reduced the time delay in the system. In [65], a secondary frequency control based on MPC for realizing aggregation and disaggregation multiple DGs was proposed, which improved the system robustness and adaptiveness. In [66], a unified model predictive voltage and current control strategy was proposed for islanded and grid-connected mode transformation, and the proposed method offered a more robust anti-harmonic capability than conventional methods. In [67], the authors proposed a decentralized model predictive control (DMPC) method for unbalanced microgrids, which has the ability to control various microgrid parameters in a predefined frequency band. Authors in [68] proposed a DMPC control method to maintain a supply/demand balance within each MG to stabilize the frequency and simultaneously achieve power exchange at the lowest cost. In [69], the authors achieved u/f compensation based on MPC in the inner level of the primary control of microgrids. Through utilizing only local variables to realize the u/f compensation with very high bandwidth, the system has a faster compensation speed than traditional MPC control methods.

A promising solution to get such features and improved performance of hierarchical coordinated control in microgrids is to combine agent-based control schemes with graph theory and dynamic consensus control. In [70], a multi-agent system (MAS) based on a finite-time global information sharing protocol including primary and secondary control was proposed to address the voltage restoration and reactive power sharing problem. This method not only achieved voltage recovery, but also ensured accurate reactive power sharing for each local DG using plug-and-play characteristics. A distributed consensus algorithm was proposed in [71] to accurately realize current and reactive power sharing among DGs in AC and DC microgrids. In [72], a finite-time consensus secondary control was proposed where the convergence time required for the proposed control strategy was not affected by the operating state of the islanded microgrid. Authors in [73] used a consensus algorithm combined with master slave control as a control strategy, significantly reducing computational complexity. Authors in [74] proposed a new consensus algorithm with better dynamic performance, which realized system regulation without the need of current information and reduced the communication and measurement dependency. A discrete consensus algorithm voltage restoration in secondary control was proposed to reduce the impact of communication noise on the consensus convergence [75]. In [20], a consensus-based distributed fixed-time secondary control was proposed, which improved the convergence speed of the current and voltage for DC microgrids. However, it is not suitable for fast changing load conditions. In [76], a small AC signal injection-based secondary frequency control was proposed to eliminate the frequency deviation by injecting an AC signal into the droop control output.

3.2.2. Application of ML on Secondary Control

Intelligence, strong scalability, and dynamic performance in the regulation of voltage and frequency in microgrids is not achieved by classical secondary control strategies [77]. Authors in [78] combined the theory of DRL and quantum ML to adaptively obtain an optimal cooperative control strategy and modify the training performance. The new control strategy achieved an effective frequency regulation, and also reduced the time delay. A secondary control based on brain emotion learning method was proposed in [79], and the algorithm was built on RL to achieve accurate compensation of the droop control. The computational effort is significantly reduced compared to conventional RL. A deep deterministic policy gradient method was proposed in [80] to improve the stability of compensation of voltage and frequency. In [81], the adaptive neuro-fuzzy inference system (ANFIS) was employed to compensate frequency and improve the speed of frequency recovery while achieving a proper power distribution. In [82], a new model-based RL algorithm based on proximal policy is proposed. Under the fast-changing system dynamics, the proposed method demonstrated superior transition capability and robustness. A neural inverse optimal distributed cooperative primary and secondary control was introduced in [83] to achieve seamless switching between islanded and grid-connected operation modes. In [84], the authors proposed an innovative online-trained ANN-based control method to realize reactive power sharing and exchange in a grid with photovoltaic and wind turbine generators, solid oxide fuel cells, and battery energy storage systems. In [85], a secondary compensating control for deep learning-based aggregation of thermostatic control loads (TCLs) is proposed to mitigate voltage imbalance. In [86], the authors proposed a new distributed secondary control scheme based on the combination of Q-learning and pining control, and built a compensation function through a greedy strategy to realize frequency and voltage compensation. It has better compensation accuracy and also allows for plug-and-play operation. In [87], the authors established several neural networks to distribute the secondary compensation based on an unsupervised ML algorithm and according to different load conditions. In [88], a platform based on Redis NoSQL (non-structured query language) database was proposed to implement a DRL-based microgrid MAS system, which provided a new idea for the implementation of ML-based microgrid control. In [89], the authors proposed secondary control of the DL algorithm based on DDPG for an islanded DC microgrid to solve the problems of voltage deviation and current sharing. In [90], a RL-based scheme was proposed for secondary frequency control. This method effectively handled general cases of resistive and inductive lines and load impedances, parameter uncertainties, time varying loads, and disturbances. No prior knowledge about the system dynamics was required once using this adaptive control approach, but how to compensate for the voltage remained in question. The proposed scheme in [91] could adjust for the control parameters adaptively, make frequency deviations to converge to a minimum through an actor-critic algorithm, and prove the convergence of the algorithm. A summary of ML-based techniques for secondary control in MGs is given in

Table 2. Summary of ML-based techniques for secondary control.

Ref.	Method	Structure	Advantages	Disadvantages	Demonstration
[78]	Multi-agent quantum deep reinforcement learning	Distributed	Achieved effective frequency regulation and reduced time delay	No stability analysis	Simulation
[79]	Brain emotional learning	Distributed	Ensures low steady-state variations with higher bandwidth and reduce calculation	The simulation only considers two DGs without considering complex scenarios	Real-time experiment
[80]	Deep deterministic policy gradient based on Q-learning	Centralized	Improve the output voltage and frequency stability	Higher requirements for communication links	Simulation
[81]	Adaptive neuro-fuzzy inference system	Distributed	Improve the speed of frequency recovery while achieving power distribution	Large amount of calculations	Simulation
[82]	A model-based reinforcement learning algorithm based on proximal policy	Distributed	Better transfer capabilities and robustness	Low computational efficiency	Simulation
[83]	Neural inverse optimal distributed cooperative	Distributed	Achieve seamless switching between islanded and grid-connected model	The effect of inconsistent line impedance is ignored	Simulation
[84]	Adaptive neural networks	Centralized	High robustness and the self-adaptation ability	Ignore communication delay	Simulation
[85]	Deep learning based artificial neural network	Distributed	Maintain system stability, hance power quality	No stability analysis	Simulation
[86]	Q-learning and pining control	Distributed	The compensation accuracy is higher and only needs compensation for pinned node nodes and plug-and-play	Over reliance on communication systems, ignore communication delay	Simulation
[87]	Unsupervised machine learning	Distributed	Improvement of accuracy of neural network model	Insufficient data samples	Simulation
[89]	DRL based DDPG	Distributed	Achieve voltage restoration and current sharing and plug-and-play	No stability analysis	Simulation
[90]	Actor-critic-based reinforcement learning	Distributed	Without a priori known dynamics of the system and compensate for the uncertain dynamics of DG	Ignore reactive power sharing problem	Simulation

. In recent years, RL algorithms have also been popular for secondary control applications, such as in [77,79,80,82,86,89,90].

Figure 4 shows a typical implementation of secondary control of microgrid based on ML techniques. As it can be seen, the agent in the secondary generates the voltage compensation value and frequency compensation value, $\Delta f$, in terms of primary control based on droop control. $s_t$ and $r_t$ are state variables and reward functions for this moment, and $s_{t+1}$ and $r_{t+1}$ are the next moment. The agent takes the underlying control loop containing the primary control as the control environment and collects the output control state parameters, such as active power, frequency, reactive power, and voltage, from this control environment. Then, through ML algorithms, it obtains the voltage and frequency compensation for the primary control and adjusts the needed compensation through the reward function until an optimal value is obtained [76].

3.3. Tertiary Control

Although the generation scale of microgrids is smaller than that of the main grids, power flow control and economic control are also indispensable [9]. Therefore, the hierarchical control strategy uses tertiary control to ensure that DGs are dispatched optimally under different operating conditions. The application objectives of tertiary control are mainly scheduling and economic related issues, such as energy management, improvement of power quality, optimal power flow, operation scheduling, and economic dispatched [4]. The optimal power exchange between the microgrid and the main grid can also be achieved in grid-connected mode. In recent years, the research in application of machine learning-based control algorithms for tertiary control has been effectively promoted and many effective control schemes have been put forward.

3.3.1. Traditional Tertiary Control

In [92], to reduce the generation cost of the hierarchical control system, authors proposed tertiary control based on an improved grey wolf algorithm to regulate active and reactive power of DG output. In [93], the authors proposed adaptive distributed averaging integral control to achieve economic optimization in tertiary control, which provides better flexibility for changing the DG output. In [94], the authors put forward a distributed-consensus-based control scheme, which combined secondary control with tertiary control. This scheme can realize economic output power of DG and voltage regulation as well. In [95], the authors used a notch filter to suppress harmonic current and achieve power reference tracking in tertiary control. In [96], the authors proposed an MPC-based aggregation scheme of energy storage systems to achieve energy management, which reduced the unnecessary calculations. In [97], in order to improve the stability of output power, a strategy of fuzzy optimized distributed cooperative control was put forward to make the system run smoothly, but the influence of transmission line impedance on the system was not considered at all. In [98], two-level layers control based on consensus algorithm was proposed to solve the optimal power flow problem and improve the overall system stability.

3.3.2. Application of ML in Tertiary Control

Data-driven methods emphasizing ML techniques to solve optimal power flow (OPF) have been presented in [99,100] and have proven to be efficient enough to address the technical challenges associated with DG uncertainties and voltage regulation. A data driven-based OPF solution for multiple DGs was presented in [99] that learns the control policies associated with each DG to substitute the solution to a centralized OPF from exclusively local information. This approach requires no manual controller tuning and little or no real-time communication. In [100], based on ML, a decentralized DG optimal dispatching method with positive dynamic characteristics has been proposed, which avoids excessive communication structures and can manage the power output better than existing control schemes. Likewise, [101] proposed an ML-based optimal control scheme considering various DGs, which includes active, reactive, and load control. In [102], the authors realized the smooth charging and discharging of energy storage unit control by NN and online RL; the new control system has high controllability and reduces disturbance. In [103], an ANN control algorithm based on autoregressive moving average has been proposed to realize power optimization regulation, which improves the stability of the system and takes into account the harmonic effect. In [104], the authors proposed a cloud-based ANN algorithm in tertiary control to reduce the operating costs. In [105], it was proposed to use a ML algorithm based on modified whale optimization for tertiary control, which enables the system to operate economically under grid-connected or islanded modes. In [106], a dynamic distributed multi-microgrid and Monte Carlo tree search-based RL was proposed for a DC microgrid to perform optimal power control. In order to realize an energy management system for cost-effective operation, a QL algorithm based on mixed integer non-linear programming was proposed in [107]. In [108], a reactive power optimization scheme based on Q-learning technology of graph convolution network was proposed, where the parameters of each agent are used to achieve more accurate operating points. In [109], the authors proposed a new RL technology, which the RL will learn from good samples rather than from a wide state space of data. Based on simulation learning, an online scheduling optimization method was proposed, which reduces the amount of calculation and complexity, improves the practicability, and reduces the operating cost of the system [110]. However, the effect of system interference is not considered. For the tertiary control of a DC microgrid, a consistency control system based on long-term and short-term deep learning was proposed in [111]. Tertiary control based on distributed RL was put forward in [112], where a learning model of the depth determination gradient algorithm was established for each DG to achieve the optimal solution of energy management. A new optimal power allocation solution was proposed in [113], which is based on a consensus algorithm and a distributed depth neural network to obtain approximate values and effectively reduce the computational burden. A summary of ML-based techniques for tertiary control in MGs is given in

Table 3. Summary of ML-based techniques for tertiary control.

Ref.	Method	Objective	Structure	Advantage	Disadvantage	Demonstration
[99]	Regression ML	Optimal power flow	Distributed	Without communication link	Disturbance is not considered	Simulation
[100]	Supervised ML support vector regression	Optimal power flow	Distributed	Avoids excessive communication structures	No stability analysis	Simulation
[101]	Supervised ML	Optimal power flow	Decentralized	Respect constraints on voltage, equipment specifications, and power capacity	Large amount of calculation	Simulation
[102]	RL based online optimal control	Optimal power flow	Distributed	Reduce the disturbances caused by C and D of various energy storage devices	Ignore communication delay	Real-time experiment
[103]	Autoregressive moving average and artificial neural networks	Optimal power flow	Distributed	Improve stability, harmonic is considered	Ignore communication delay	Simulation
[104]	A cloud-based ML using an artificial neural network	Islanded detection	Centralized	Less data traffic, decrease the cost	The operation results under complex conditions are not considered	Real-time experiment
[105]	ML based on modified whale optimization	Optimal power flow	Centralized	More economic	Disturbance is not considered	Simulation
[106]	Monte Carlo tree search-based reinforcement learning	Optimal power flow	Centralized	Maintaining system security constraints while solving economic problems	Disturbance is not considered	Simulation
[107]	MINLP-QL	Energy management	Centralized	Reduce costs and calculations	Large amount of calculation	Simulation
[108]	Based on GCN and deep Q-learning	Optimal power flow	Centralized	Achieves more accurate reactive power compensation and better voltage stability	Ignore communication delay	Simulation
[109]	A novel RL technique based on classical recurrent neural networks	Energy management	Centralized	Fast learning systems with a small number of training samples	Uncertainty and interference are not considered	Simulation
[110]	A novel imitation learning based online scheduling	Optimal power flow	Centralized	Reduced complexity and computational effort, increased efficiency	Uncertainty and interference are not considered	Simulation
[111]	Long short-term memory deep learning models	Energy management	Distributed	Stabilize the bus voltages and achieve higher endurability	Large amount of calculation	Real-time experiment
[112]	Deep reinforcement learning	Energy management	Distributed	Reduce the computational complexity, avoids the leakage of private keys	Ignore communication delay	Simulation
[113]	Deep neural network	Optimal power flow	Distributed	Take less computational time and achieve real-time optimization	Ignore energy storage system	Simulation

.

3.4. Communication Network

In the above control methods, both centralized control (as shown in Figure 5) and distributed control (as shown in Figure 6) schemes require information exchange [5,114]. Such information exchange is scheduled according to the specified communication cycles. For distributed control, each agent needs to update its status information to its neighboring nodes at each iteration. This also makes the use of communication networks more conservative [9]. When the control messages are passed between DGs, delays inevitably occur, both in high and low bandwidth communication networks [12,114]. In addition, interference and data packet loss during communication are also influential factors that must be considered. Especially when multi-objective control is implemented, the communication network is prone to delay, packet loss, and other phenomena, which lead to system stability weakening or even failure in some consensus.

In order to reduce the communication burden, a distributed frequency and voltage secondary control based on the finite time event-triggered consensus (FETC) scheme was proposed in [115]. This scheme combined the characteristics of event triggering and limited time response and was verified by experiments. In [116], the authors proposed a secondary control based on an event trigger device, which takes into account frequency recovery and economic dispatching, and sets the combination of the two to be executed at an hourly scale, thus reducing communication times and costs. In [117], the authors proposed a simplified distributed event-triggered secondary control without extra state estimators, which simplifies the design parameters and avoids periodic communication. In [118], a delay compensated control based on the Artstein model is proposed, which mitigates the communication delay but ignores the voltage deviation generated by the system. In [55], the authors proposed a delay minimization scheme based on Q-learning, which minimizes the delay while allocating resources and does not require much experience. This scheme has less latency and high fairness. In [119], the authors proposed a control scheme of multi-agent Bayesian DRL to realize communication among agents. In order to reduce the impact of communication failure, the authors also proposed a new Bayesian update method based on which other agent behaviors can be estimated in the event of a failure.

3.5. Analysis and Technical Summary

The analysis presented above demonstrates the significant achievements of ML techniques in microgrid hierarchical control. ML-based control schemes exhibit superior dynamic characteristics compared to traditional approaches, enabling accurate compensation and faster response times during load fluctuations. As shown in [80], the control parameters were self-adjusting. The need for a priori data is eliminated in many systems [90]. The predictive capability of ML techniques greatly enhances control performance. Moreover, ML techniques can enhance system robustness and stability, as evidenced in [48,85,103]. However, there are several technical challenges associated with ML techniques. These include the computational complexity and data processing requirements, as well as the establishment of appropriate reward functions. Furthermore, there is limited research on the stability analysis of hierarchical control using ML technology, and validation experiments with physical support are scarce. Additional investigations are necessary to validate the reliability and effectiveness of ML technology in microgrid control and operation.

The distribution of ML algorithms utilized in microgrid hierarchical control is illustrated in Figure 7. It is observed that supervised learning and RL are the two primary approaches. Among the literature employing supervised learning, 61% utilizes neural network-based control, indicating that neural networks are currently a prominent research area. Regarding RL, 35% of studies are based on Q-learning, while 56% focus on DRL, both of which are considered mainstream research directions. Notably, recent research has shown increased attention towards combining Q-learning with DDPG or MINLP, as well as the application of DRL with DDPG. However, it is important to address the challenges related to computational requirements and communication issues associated with these approaches.

Furthermore, since the existence of renewable energy sources such as solar and wind energy is influenced by the environment, ML techniques are widely used to predict their generation capacity and can also be used to predict the consumption of some unstable users to optimize their energy consumption. Based on ML, [120] proposed solar-based power generation prediction for complex scenarios and [121] proposed wind-based power generation prediction. ML can be used to optimize the use of ESS in microgrids to ensure that energy is stored and used efficiently. In addition, ML can be used to optimize the use of ESS in microgrids to ensure that energy is stored and used efficiently. The optimal use of energy storage modules based on ML was given in [30]. By analyzing the ESS operation data, ML can help the user to set the optimal schedule for charging and discharging the ESS while considering the energy price and the health of the energy storage unit. ML techniques are also used to build models to analyze system output parameters and data obtained by sensors in the environment to detect potential faults and anomalies in the system and avoid safety problems in the system [104]. In addition, ML techniques can also be used for energy trading between microgrid participants by analyzing historical trading data and price markets to enable self-help trading between multiple users and reduce operating costs. In [122], the authors proposed an RL-based scheme to optimize trading decisions. In terms of communication network security, protection mechanisms can also be established using ML techniques to achieve identification of abnormal network states and active system protection. In conclusion, ML technology has good prospects for application in energy and load forecasting, energy storage optimization, fault detection, energy trading, and communication security.

4. Discussion and Future Scopes

Droop-based primary control using PI controllers has been widely used in islanded/grid-connected microgrids, but the control parameters may not be precise enough to assure optimality of the operation in different working conditions. Based on this outcome, ML control methods based on neural networks and RL have achieved good control results in parameter selection, but there are still research gaps. There are limited quality datasets available for selection and the data reliability needs to be improved, so it is difficult to maintain the high accuracy of the training and validation process. Simultaneously, there is no standard set of criteria for the selection of machine learning algorithms for different control requirements. Furthermore, the interpretability of its control process and the accountability of various control levels are not defined. It is also a major difficulty to establish a mathematical model for each level of control of the hierarchical control to simulate the behavior of the microgrid under different operating conditions. The results of the literature review show that stability analysis of the ML algorithms to ensure a stable operation of the system when it is disturbed still needs to be analyzed and studied [10]. In addition, under the complex operating conditions with multiple DGs, issues related to data explosion and complex calculations, and in real-time microgrid control, it is important to balance model accuracy and computational performance.

For the secondary control system, compensation methods using consensus algorithm and PI controllers have been widely studied by scholars; however, the traditional control methods are difficult to effectively control the complex microgrid configurations, especially when the system size gets larger with many control parameters [123]. ML-based techniques can be a good choice to address the secondary compensation issues. Based on the literature review, some papers give only active frequency control or reactive voltage control based on ML. However, how to implement both kinds of control based on the proposed algorithm and establish a complete microgrid hierarchical control system based on ML algorithm needs to be studied. In addition, the communication link is also a research topic that cannot be ignored. For microgrid hierarchical control systems, the standardization of communication protocols and communication structures needs to be enhanced, and how to select message intervals and communication frame sizes need to be standardized. Consequently, it is difficult to achieve optimality in terms of communication resource scheduling allocation for different control functions such as monitoring or protection. Moreover, the optimization problem in achieving the balance between communication requirements and energy consumption is not sufficiently studied. Preventing communication attacks, protecting private security, obtaining communication redundancy, solving communication packet loss, and improving the robustness of communication systems are also research hotspots. The control scheme based on RL can effectively simplify the communication system, but its stability still needs to be analyzed. This is because there is always the risk that the RL algorithm does not ensure accurate prediction of the system behavior under any conditions. Furthermore, the complexity of the operating system can be influenced by the environment and is full of uncertainty, for example, wind- and solar-based power generation systems can change dynamically with the weather [25]. Furthermore, control systems are at risk of cyber-attacks, which also causes uncertainty in the control system [16]. In a multi-microgrid system, in parallel to renewable energy variability, there are also load demand instabilities, as well as equipment failure issues and communication and control uncertainties to consider. The traditional control methods used for optimizing power flow and economic dispatch in tertiary control systems have shortcomings such as low computational efficiency and poor convergence speed. While ML-based control methods give better solutions. In [112,113], it was evident that distributed control methods for DNN-based ML give better solutions, which are more economical in terms of results, and simplify the complexity of the algorithm and reduce the training time compared to the traditional control scheme. RL and DNN provide a new concept in solving the inertia control problem of microgrids [124,125]. The use of algorithms based on a combination of ANN and DRL is an emerging trend in research. How to accurately plan generation and analyze the impact of load changes on tertiary control is also an important scope for future research [126].

How to establish an effective control model at each control level of hierarchical control is a further challenge. In hybrid AC–DC microgrid systems, the lack of effective control strategies, especially in multi-energy system conditions including fuel cells, will make the control environment more sophisticated and new control strategies need to be proposed. Additionally, combining ML techniques with other artificial intelligence strategies is needed to improve control efficiency, especially in online decision making. The internet of things (IoT) technology and distributed ledger can be adopted to promote the information exchange capability and security control of microgrid systems, based on which new communication protocols can be explored and new communication standards can be proposed. Additionally, the economic performance of microgrids can be improved by making them participate in the autonomous market and allowing them to support each other with other microgrids. A large amount of valid data is required for training ML models [127], and how to validate the original data is also an important area to be studied. In addition, from the literature review, it can be seen that most scholars only validate the results at the simulation level (Table 1, Table 2 and Table 3), and how to implement a hardware platform to realize the effectiveness of the proposed control strategy is also one of the problems to be solved.

5. Conclusions

This paper has reviewed the microgrid hierarchical control literature that has been published in the past five years, mainly by analyzing the application of ML in each level of microgrid hierarchical control systems and outlining the shortcomings of traditional control methods. A specific comparative analysis of ML applications for each control layer was given, and ML-based primary and secondary control schematics were summarized based on previous studies. The literature review showed that ML empowers microgrids with predictive insights and intelligent control and optimization. It can be used in a wide range of applications from forecasting renewable energy, managing energy storage options, enhancing the grid stability, enabling demand response, detecting faults, to optimizing the operations, fortifying cybersecurity, and more. With data-driven precision, microgrids become resilient, efficient, and sustainable, revolutionizing the way we manage and harness energy. It was also observed that ML-based control methods have become one of the main research directions in microgrid hierarchical control systems, and the proportion of research applying ML to hierarchical control is increasing, especially in the field of ANN and improved algorithms based on DRL, QL, etc. One advantage of RL is its ability to reduce reliance on priori data. However, attention should also be directed towards acquiring accurate and valid data to support the application of supervised and unsupervised techniques, as high-quality data plays a crucial role in improving predictions. Additionally, exploring semi-supervised learning techniques in microgrid hierarchical control remains limited, and further investigations in this area could provide valuable insights. Stability analysis is another critical concern associated with using ML technology in microgrid control, particularly with the increasing number of DG and ESS installations. Extensive research is necessary to address stability issues effectively. In terms of microgrid communication, although some progress has been made, existing schemes still face challenges in achieving smooth control, and there is a lack of communication standards for ML technology applications. It is important to address these issues to ensure reliable and efficient communication within microgrids.

Furthermore, there is a need for further exploration of computational efficiency, experimental validation, and verification of ML techniques in microgrid control. These areas require dedicated research efforts to overcome the associated difficulties.