A Novel Deep Reinforcement Learning-Based Current Control Method for Direct Matrix Converters

Abstract

This paper presents the first approach to a current control problem for the direct matrix converter (DMC), which makes use of the deep reinforcement learning algorithm. The main objective of this paper is to solve the real-time capability issues of traditional control schemes (e.g., finite-set model predictive control) while maintaining feasible control performance. Firstly, a deep Q-network (DQN) algorithm is utilized to train an agent, which learns the optimal control policy through interaction with the DMC system without any plant-specific knowledge. Next, the trained agent is used to make computationally efficient online control decisions since the optimization process has been carried out in the training phase in advance. The novelty of this paper lies in presenting the first proof of concept by means of controlling the load phase currents of the DMC via the DQN algorithm to deal with the excessive computational burden. Finally, simulation and experimental results are given to demonstrate the effectiveness and feasibility of the proposed methodology for DMCs.

1. Introduction

The direct matrix converter (DMC) is a promising topology due to its numerous advantages, such as sinusoidal input and output currents, controllable input power factor, and compact design without a DC-link capacitor [1,2,3]. These prominent features make the DMC an alternative to the traditional back-to-back converter in various industrial applications where size and lifetime are critical issues.

In the past few decades, numerous modulation and control methods for DMCs have been introduced in the literature, among which the space vector modulation (SVM) has gained the most popularity for its inherent capability to track both the reference output voltage vector and input current vector simultaneously [4,5]. However, with the rapid development of digital processors and power devices, the SVM is now being challenged by the model predictive control (MPC) due to its simpler theoretical complexity, easier implementation, and better dynamic response [6,7,8]. The MPC method involves solving a finite-horizon optimization problem at each time step by predicting future system behavior, optimizing a cost function and applying only the first control input of the sequence. Although the MPC has been considered an emerging alternative to the traditional SVM, the computational burden of solving the optimization problem, the accurate modeling of system dynamics and constraints, and the selection of appropriate cost functions are well-known obstacles to its real-world applications [9,10,11,12,13,14,15,16]. The first major challenge is the computational complexity associated with solving the optimization problem due to the fast switching frequencies and complex dynamics of the system. This can result in high processing times and control delays. To address this challenge, various approaches have been proposed, such as reduced-order models and optimization algorithm improvements [9,10,11]. Another critical challenge is the precise modeling of the system dynamics and constraints, which are typically complex and nonlinear. Researchers have proposed adaptive and robust methods that can account for uncertainties and modeling errors in real time [12,13,14]. Additionally, the selection of an appropriate cost function for the MPC controller is crucial, as it affects the control performance, energy efficiency, and system stability. Recent research has focused on developing new cost functions that can balance these competing objectives more effectively [15,16]. Addressing these challenges is crucial for the continued development and application of MPC in power electronics, and ongoing research is focused on developing new and improved methods to overcome these issues.

Recently, the fast growth of artificial intelligence technology has changed the traditional control strategy of the past few decades [17,18,19]. Reinforcement learning (RL) is a subfield of machine learning concerned with how an agent can learn to take actions that maximize a cumulative reward signal in an uncertain environment. RL is a powerful approach for building intelligent systems that can learn from experience and make decisions based on complex and dynamic inputs. In recent years, there has been growing interest in RL as a result of its success in a wide range of domains, from playing complex games such as Go and chess to controlling complex robotic systems. RL has also shown promise in addressing real-world problems, such as optimizing energy consumption and navigating autonomous vehicles [20,21,22,23]. In contrast to the MPC, RL agents try to find the optimal control policy during the training process before their real-world implementation, which makes it possible to avoid the computationally costly online optimization in each sampling period. Furthermore, the RL control method can be trained in field applications to take parameter variations and parasitic effects into account. As a result, RL has become an active research area with many ongoing studies exploring new algorithms, applications, and theoretical foundations.

Motivated by the aforementioned shortcomings of the MPC method and the superiority of the RL method, the potential of utilizing RL methods in power electronics is being explored [24,25,26,27]. Deep Q-Network (DQN) is a type of RL algorithm that uses a neural network to approximate the Q-function, which estimates the expected return for taking a particular action in a given state. DQN has shown promising results in various application scenarios with continuous states and discrete actions [28,29]. Although the DQN algorithm has emerged as a promising method for controlling power electronics systems, it faces significant challenges that need to be addressed. One of the primary challenges of using DQN in power electronics is the issue of high-dimensional state and action spaces. This can make it difficult to train the neural network effectively and can result in slow convergence and poor performance. Another challenge is the stability of DQN during training. DQN can suffer from issues such as overfitting, instability, and divergence, which can result in poor performance or even catastrophic failure of the controller. Addressing this challenge requires developing methods for stabilizing DQN during training, such as target network updating, experience replay, and parameter initialization. Due to the aforementioned challenges, no attempt has been made to incorporate the DQN algorithm with the DMC system.

In view of the above observations, this paper is concerned with a novel approach to the current control problem for the DMC, which makes use of the DQN algorithm. Specifically, an agent is trained without any plant-specific knowledge to find the optimal control policy by direct interaction with the system. Thus, the online optimization process is carried out in advance. The main merit of this proposal is that the computational burden problems can be alleviated by deploying the proposed solution. Furthermore, the proposal can be easily expanded to different power converters with finite switching states. Finally, the performance evaluation of the proposed methodology for DMCs in comparison to the state-of-the-art finite control set model predictive current control approach is given to confirm the effectiveness and feasibility of the proposal.

We contribute two main points to the relevant literature. (1) To the best of the authors’ knowledge, this is the first time the DQN algorithm is incorporated with the current control method of the DMC. (2) Another important contribution of this paper is that the heavy computational burden can be reduced dramatically by the utilization of the trained agent so as to carry out the optimization problem in the training phase in advance, which allows for low-cost processors.

2. Proposed DQN-Based Current Control Method for DMC

The common topology of a three-phase DMC is shown in Figure 1, which consists of nine bi-directional switches to connect the input voltage source to the output load. An input filter ($L_{\mathrm{i}}$, $R_{\mathrm{i}}$, $C_{\mathrm{i}}$) is installed to eliminate high-frequency harmonics of the input current and reduce the input voltage distortion supplied to the DMC. The DMC performs AC/AC power conversion in a single stage, while the indirect matrix converters (IMC) achieve this in two stages, namely, rectification and inversion stages. The implementation of DMC requires 18 reverse-blocking IGBTs while the IMC consists of 12 reverse-blocking IGBTS and 6 reverse-conduction IGBTs. In comparison, the virtual DC-link stage of IMC makes it easier to construct with fewer switches, such as the sparse matrix converter, which is beyond the scope of this paper.

According to Figure 1, the instantaneous relationship between the input and output quantities can be described as

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{u}_{\mathrm{o}\mathrm{A}}\hfill \\ u_{\mathrm{o}\mathrm{B}}\hfill \\ u_{\mathrm{o}\mathrm{C}}\hfill \end{array}\right]& =\left[\begin{array}{ccc}{S}_{\mathrm{a}\mathrm{A}}\hfill & S_{\mathrm{b}\mathrm{A}}\hfill & S_{\mathrm{c}\mathrm{A}}\hfill \\ S_{\mathrm{a}\mathrm{B}}\hfill & S_{\mathrm{b}\mathrm{B}}\hfill & S_{\mathrm{c}\mathrm{B}}\hfill \\ S_{\mathrm{a}\mathrm{C}}\hfill & S_{\mathrm{b}\mathrm{C}}\hfill & S_{\mathrm{c}\mathrm{C}}\hfill \end{array}\right]\left[\begin{array}{c}{u}_{\mathrm{e}\mathrm{a}}\hfill \\ u_{\mathrm{e}\mathrm{b}}\hfill \\ u_{\mathrm{e}\mathrm{c}}\hfill \end{array}\right]\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{i}_{\mathrm{e}\mathrm{a}}\\ i_{\mathrm{e}\mathrm{b}}\\ i_{\mathrm{e}\mathrm{c}}\end{array}\right]& =\left[\begin{array}{ccc}{S}_{\mathrm{a}\mathrm{A}}\hfill & S_{\mathrm{a}\mathrm{B}}\hfill & S_{\mathrm{a}\mathrm{C}}\hfill \\ S_{\mathrm{b}\mathrm{A}}\hfill & S_{\mathrm{b}\mathrm{B}}\hfill & S_{\mathrm{b}\mathrm{C}}\hfill \\ S_{\mathrm{c}\mathrm{A}}\hfill & S_{\mathrm{c}\mathrm{B}}\hfill & S_{\mathrm{c}\mathrm{C}}\hfill \end{array}\right]\left[\begin{array}{c}{i}_{\mathrm{o}\mathrm{A}}\hfill \\ i_{\mathrm{o}\mathrm{B}}\hfill \\ i_{\mathrm{o}\mathrm{C}}\hfill \end{array}\right]\hfill \end{array}$$

(2)

where $u_{\mathrm{o}\mathrm{A}}$, $u_{\mathrm{o}\mathrm{B}}$, $u_{\mathrm{o}\mathrm{C}}$ and $u_{\mathrm{ea}}$, $u_{\mathrm{e}\mathrm{b}}$, $u_{\mathrm{e}\mathrm{c}}$ are the output and input phase voltages of the DMC, $i_{\mathrm{oA}}$, $i_{\mathrm{oB}}$, $i_{\mathrm{oC}}$ and $i_{\mathrm{ea}}$, $i_{\mathrm{eb}}$, $i_{\mathrm{ec}}$ are the output and input currents of the DMC, respectively, $S_{\mathrm{xy}}=1$ with $\mathrm{x}\in (\mathrm{a},\mathrm{b},\mathrm{c})$ and $\mathrm{y}\in (\mathrm{A},\mathrm{B},\mathrm{C})$ means the switch is on while $S_{\mathrm{xy}}=0$ means the switch is off.

For safe operation, the input phases should not be short-circuited, and the load should not be open-circuited. Thus, the switching constraints of the DMC can be expressed as

$$\left\{\begin{array}{c}{S}_{\mathrm{a}\mathrm{A}}+S_{\mathrm{b}\mathrm{A}}+S_{\mathrm{c}\mathrm{A}}=1\hfill \\ S_{\mathrm{a}\mathrm{B}}+S_{\mathrm{b}\mathrm{B}}+S_{\mathrm{c}\mathrm{B}}=1\hfill \\ S_{\mathrm{a}\mathrm{C}}+S_{\mathrm{b}\mathrm{C}}+S_{\mathrm{c}\mathrm{C}}=1\hfill \end{array}\right..$$

(3)

Therefore, there are 27 valid switching states for the DMC.

The basic RL setting consists of an agent and environment. At each time step k, the agent observes the current state ${\mathbf{O}}_{\mathrm{k}}$ of the environment, and an action $A_{\mathrm{k}}$ is taken according to the policy $\pi$. Based on ${\mathbf{O}}_{\mathrm{k}}$ and $A_{\mathrm{k}}$, the environment is updated to ${\mathbf{O}}_{\mathrm{k}+1}$, and a reward $R_{\mathrm{k}}$ is produced, both of which are received by the agent. The observation–action–reward cycle continues until the training process is complete. The goal of the agent is to use RL algorithms to learn the best policy as it interacts with the environment so that given any state, it will always take the most optimal action that produces the most reward in the long run [30]. The action-value function $Q^{\pi }\left({\mathbf{O}}_{\mathrm{k}},A_{\mathrm{k}}\right)$ is introduced to evaluate the expected cumulative discounted reward as [28]

$$\begin{array}{cc}\hfill & Q^{\pi }\left({\mathbf{O}}_{\mathrm{k}},A_{\mathrm{k}}\right)=\mathbb{E}\left\{\sum _{i=k}^{\infty }{\gamma }^{i-k}{R}_{\mathrm{i}}\mid \mathbf{O}={\mathbf{O}}_{\mathrm{k}},A=A_{\mathrm{k}}\right\}\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left\{R_{\mathrm{k}}+\gamma Q^{\pi }\left({\mathbf{O}}_{\mathrm{k}+1},A_{\mathrm{k}+1}\right)\mid \mathbf{O}={\mathbf{O}}_{\mathrm{k}},A=A_{\mathrm{k}}\right\}\hfill \end{array}$$

(5)

where $\gamma \in [0,1)$ is the discount factor allowing the control task to be adjusted from short-sighted to far-sighted, and $\mathbb{E}\{·\}$ denotes the expected value.

In the DMC, the observation consists of the measured input phase voltage ($u_{e\alpha }$, $u_{e\beta }$), output load current ($i_{o\alpha }$, $i_{o\beta }$), and the errors between the measured and the reference load current ($\Delta i_{o\alpha }$, $\Delta i_{o\beta }$), which looks as follows:

$$\begin{array}{cc}\hfill \mathbf{O}=& \left[u_{e\alpha },u_{e\beta },i_{o\alpha },i_{o\beta },\Delta i_{o\alpha },\Delta i_{o\beta }\right].\hfill \end{array}$$

(6)

According to the constraints in Equation (3), when only one zero switching state is included, the action space A contains 25 options, which can be defined as

$$A=\left\{S_0,S_1,S_2,\dots ,S_{24}\right\}.$$

(7)

To improve the policy of the agent with trial and error, an appropriate reward function should be designed. In this paper, the DMC should operate with the load current accurately following the reference value. Thus, the reward function is defined as

$$R=-(\Delta i_{o\alpha }^2+\Delta i_{o\beta }^2).$$

(8)

The reference value of the load current is given as

$${\mathit{i}}_{\mathrm{o}}^{\ast }=\left[I_{\mathrm{o}\mathrm{m}}^{\ast }cos{\varphi }_{\mathrm{o}}{I}_{\mathrm{o}\mathrm{m}}^{\ast }cos({\varphi }_{\mathrm{o}}-2\pi /3)I_{\mathrm{o}\mathrm{m}}^{\ast }cos({\varphi }_{\mathrm{o}}+2\pi /3)\right]$$

(9)

where ${\varphi }_{\mathrm{o}}$ is the expected angle of the load current and $I_{\mathrm{o}\mathrm{m}}^{\ast }$ is the amplitude of the expected load current.

According to Equations (4) and (5), the expectable return is represented by the action-value function based on the state–action pair at each time step. To maximize the expected cumulative reward over time, a new policy $\pi {}^{\prime }$ better than $\pi$ can be found as

$${\pi }^{\prime }\left({\mathbf{O}}_{\mathrm{k}}\right)=arg\underset{A}{max}{Q}^{\pi }({\mathbf{O}}_{\mathrm{k}},A)$$

(10)

Thus, one major challenge in the DQN algorithm is to derive an accurate mapping from state–action pairs to values. With the help of the neural network, the $Q^{\pi }({\mathbf{O}}_{\mathrm{k}},A_{\mathrm{k}})$ can be estimated by a universal function approximator $Q_{\theta }^{\pi }({\mathbf{O}}_{\mathrm{k}},A_{\mathrm{k}})$ with weights and biases (critic parameters) represented by $\theta$. The network has four layers: an input layer, two hidden layers, and an output layer. The hidden layers are fully connected, and the ReLU function is adopted as the activation function.

To train the network, state transition experiences ${\mathcal{E}}_{\mathrm{i}}=\left\{{\mathbf{O}}_{\mathrm{i}},A_{\mathrm{i}},R_{\mathrm{i}},{\mathbf{O}}_{\mathrm{i}+1}\right\}$ are stored in the experience buffer, from which a random mini-batch $\mathcal{M}$ of M experiences is sampled to update $\theta$ by reformulating the Bellman equation in Equation (5) as a minimization problem of the loss $L_Q$:   

$$\begin{array}{cc}\hfill & \underset{\theta }{min}{L}_Q\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.L_Q=\frac{1}{M}\sum _{{\mathcal{E}}_{\mathrm{i}}\in \mathcal{M}}{\left(Q_{\theta }^{\pi }({\mathbf{O}}_{\mathrm{i}},A_{\mathrm{i}})-\left(R_{\mathrm{i}}+\gamma \underset{A}{max}{Q}_{{\theta }_{\mathbf{t}}}^{\pi }({\mathbf{O}}_{\mathrm{i}+1},A)\right)\right)}^2\hfill \end{array}$$

(11)

where $Q_{{\theta }_{\mathbf{t}}}^{\pi }({\mathbf{O}}_{\mathrm{i}},A_{\mathrm{i}})$ is the target critic, which improves the stability of the bootstrapping methods. The parameters ${\theta }_{\mathbf{t}}$ of the target network are updated periodically:

$${\theta }_t\leftarrow \theta ,\mathrm{after}\mathrm{every}{N}_{\mathrm{T}}\mathrm{steps}.$$

(12)

At last, the tradeoff between exploration and exploitation is performed to avoid the learning algorithm converging into a suboptimal policy. Therefore, the $ϵ$-greedy policy is introduced as

$$A_{\mathrm{k}}=\left\{\begin{array}{cc}\underset{A}{argmax}{Q}_{\theta }^{\pi }({\mathbf{O}}_{\mathrm{k}},A),\hfill & \mathrm{with}\mathrm{probability}1-ϵ\hfill \\ \mathrm{a}\mathrm{random}\mathrm{element}\mathrm{from}A,\hfill & \mathrm{with}\mathrm{probability}ϵ\hfill \end{array}\right.$$

(13)

where $ϵ$ updates at the end of each training step:

$$ϵ=ϵ·(1-{ϵ}_{\mathrm{decay}}).$$

(14)

Note that $ϵ$ is set to zero when the training process has been completed. The schematic of the overall control structure with a learning routine is presented in Figure 2, and the learning pseudocode is given in Algorithm 1.

Algorithm 1: DQN pseudocode

Initialize the critic $Q_{\theta }^{\pi }(\mathbf{O},A)$ with random parameter values $\theta$. Initialize the target critic $Q_{{\theta }_{\mathbf{t}}}^{\pi }(\mathbf{O},A)$ with parameters: ${\theta }_{\mathbf{t}}=\theta$. $\mathbf{for}$ episode=1 to max-episode $\mathbf{do}$:            Observe the initial state ${\mathbf{O}}_0$.    $\mathbf{for}$ step=1 to max-step $\mathbf{do}$:                1. For the current observation ${\mathbf{O}}_{\mathrm{k}}$, select the action $A_{\mathrm{k}}$ based on Equations (13) and (14).                2. Execute action $A_{\mathrm{k}}$. Observe the next observation ${\mathbf{O}}_{\mathrm{k}+1}$ and reward $R_{\mathrm{k}}$.                3. Store (${\mathbf{O}}_{\mathrm{k}}$, $A_{\mathrm{k}}$, $R_{\mathrm{k}}$, ${\mathbf{O}}_{\mathrm{k}+1}$) in the experience buffer.                4. Sample a random mini-batch of experiences (${\mathbf{O}}_{\mathrm{i}}$, $A_{\mathrm{i}}$, $R_{\mathrm{i}}$, ${\mathbf{O}}_{\mathrm{i}+1}$) from the experience buffer.                5. Update the critic parameters using Equation (11).                6. Update the target critic parameters using Equation (12).                7. Reset the environment and break if ${\mathbf{O}}_{\mathrm{k}+1}$ is the terminal state.    $\mathbf{end}\mathbf{for}$ $\mathbf{end}\mathbf{for}$

3. MPC Method for DMC

First, the input filter model is established for the prediction of input voltages and currents. In this paper, the LC filter with a damping resistor is adopted, as shown in Figure 3.

The continuous system model of the input filter in Figure 3 can be described by the following equations:

$$\begin{array}{cc}\hfill \left[\begin{array}{c}\frac{d{\mathit{u}}_e}{dt}\hfill \\ \frac{d{\mathit{i}}_L}{dt}\hfill \end{array}\right]& =A\left[\begin{array}{c}{\mathit{u}}_e\hfill \\ {\mathit{i}}_L\hfill \end{array}\right]+B\left[\begin{array}{c}{\mathit{u}}_s\hfill \\ {\mathit{i}}_e\hfill \end{array}\right]\hfill \\ & =\left[\begin{array}{cc}-\frac{1}{R_{in}{C}_{in}}& \frac{1}{C_{in}}\\ -\frac{1}{L_{in}}& 0\end{array}\right]\left[\begin{array}{c}{\mathit{u}}_e\hfill \\ {\mathit{i}}_L\hfill \end{array}\right]+\left[\begin{array}{cc}\frac{1}{R_{in}{C}_{in}}& -\frac{1}{C_{in}}\\ \frac{1}{L_{in}}& 0\end{array}\right]\left[\begin{array}{c}{\mathit{u}}_s\hfill \\ {\mathit{i}}_e\hfill \end{array}\right]\hfill \end{array}$$

(15)

where $L_{in}$, $C_{in}$, and $R_{in}$ are the filter inductance, the filter capacitance, and the filter damping resistance, respectively.

A discrete state space model can be derived when a forward Euler approximation is applied to a continuous-time system described in the state space form of Equation (15). Considering a sampling period $T_s$, the discrete-time input filter model can be described as

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\mathit{u}}_e(k+1)\hfill \\ {\mathit{i}}_L(k+1)\hfill \end{array}\right]& =G\left[\begin{array}{c}{\mathit{u}}_e\left(k\right)\\ {\mathit{i}}_L\left(k\right)\end{array}\right]+H\left[\begin{array}{c}{\mathit{u}}_s\left(k\right)\hfill \\ {\mathit{i}}_e\left(k\right)\hfill \end{array}\right]\hfill \\ & =\left[\begin{array}{cc}{G}_{11}\hfill & G_{12}\hfill \\ G_{21}\hfill & G_{22}\hfill \end{array}\right]\left[\begin{array}{c}{\mathit{u}}_e\left(k\right)\hfill \\ {\mathit{i}}_L\left(k\right)\hfill \end{array}\right]+\left[\begin{array}{cc}{H}_{11}\hfill & H_{12}\hfill \\ H_{21}\hfill & H_{22}\hfill \end{array}\right]\left[\begin{array}{c}{\mathit{u}}_s\left(k\right)\hfill \\ {\mathit{i}}_{\mathbf{e}}\left(k\right)\hfill \end{array}\right]\hfill \end{array}$$

(16)

where $G=e^{A{T}_s}$, $H=A^{-1}(G-I)B$. Using Equation (16), the value of ${\mathit{u}}_e$ and ${\mathit{i}}_L$ in the next sampling instant can be predicted.

The model of the resistance–inductance load is given by

$$\frac{d{\mathit{i}}_o}{dt}=\frac{1}{L_o}\left({\mathit{u}}_o-R_o{\mathit{i}}_o\right)$$

(17)

where $L_o$ and $R_o$ are the inductance and resistance of the load.

Similarly, using the forward Euler approximation, the equation for the load current prediction can be derived as

$${\mathit{i}}_o(k+1)=\left(1-\frac{R_o}{L_o}{T}_s\right)·{\mathit{i}}_o\left(k\right)+\frac{T_{\mathrm{s}}}{L_o}{\mathit{u}}_o\left(k\right)$$

(18)

For 27 different switching states of the DMC, the corresponding load voltage vector ${\mathit{u}}_o\left(k\right)$ and input current vector ${\mathit{i}}_e\left(k\right)$ are calculated to predict the value of ${\mathit{i}}_o(k+1)$, ${\mathit{i}}_L(k+1)$, and ${\mathit{u}}_e(k+1)$ in the next sampling interval. The source current ${\mathit{i}}_s(k+1)$ is calculated by

$${\mathit{i}}_s(k+1)=\frac{{\mathit{u}}_s(k+1)-{\mathit{u}}_e(k+1)}{R_{in}}+{\mathit{i}}_L(k+1).$$

(19)

The current control objectives of the DMC are twofold: to regulate the grid-side current ${\mathit{i}}_s$ for unit power factor operation and to adjust the output current ${\mathit{i}}_o$ for symmetrical and sinusoidal three-phase load current. The reference values for ${\mathit{i}}_o$ are the same as Equation (9), and ${\mathit{i}}_s$ and are defined as follows:

$$i_s^{\ast }={\left[\begin{array}{ccc}{I}_{sm}^{\ast }cos{\phi }_{in}\hfill & I_{sm}^{\ast }cos\left({\phi }_{in}-2\pi /3\right)\hfill & I_{sm}^{\ast }cos\left({\phi }_{in}+2\pi /3\right)\hfill \end{array}\right]}^T$$

(20)

where ${\phi }_{in}$ and $I_{sm}^{\ast }$ are the expected phase angle and amplitude of the source current.

The errors of the predicted source current $i_s^P$ and load current $i_o^P$ in static two-phase coordinates can be expressed as

$$\Delta i_s={\left(\Delta i_{s\alpha }\Delta i_{s\beta }\right)}^T=T_{abcto\alpha \beta }\left(i_s^P-i_s^{\ast }\right)$$

(21)

$$\Delta i_o={\left(\Delta i_{o\alpha }\Delta i_{o\beta }\right)}^T=T_{abcto\alpha \beta }\left(i_o^P-i_o^{\ast }\right)$$

(22)

where

$$T_{\mathrm{abcto}\alpha \beta }=\frac{2}{3}\left[\begin{array}{ccc}1& -\frac{1}{2}& -\frac{1}{2}\\ 0& \frac{\sqrt{3}}{2}& -\frac{\sqrt{3}}{2}\end{array}\right]$$

(23)

The cost function is designed to penalize differences from the reference value:

$$\left\{\begin{array}{c}g=\lambda \ast g_1+g_2\hfill \\ g_1=\Delta i_{s\alpha }^2+\Delta i_{s\beta }^2\hfill \\ g_2=\Delta i_{o\alpha }^2+\Delta i_{o\beta }^2\hfill \end{array}\right.$$

(24)

where $\lambda$ is the weighting factor for the source current control. In this paper, the DQN method is trained to focus on the output current. Thus, for a fair comparison, $\lambda$ is set to 0. In practice, $\lambda =1$ provides a fairly good load current in comparison to $\lambda =0$ due to the fact that ${\mathit{u}}_e$ is controlled to be more sinusoidal.

In each sampling period, all 27 possible switching states are used to calculate the cost function, and the switching state corresponding to the minimum value of the cost function is applied to the DMC in the next sampling time.

In practical applications, due to the delay of the digital controller, the switching state selected at a certain moment can only be applied to the converter in the next moment, and the switching state applied at that moment may not be the optimal one for the next moment, which may result in significant errors. In order to make the selected optimal switching state act on the converter at a reasonable time, a two-step prediction strategy is usually adopted. The specific implementation process is as follows: based on the sampled value of the current system state $x\left(k\right)$, predict the value of the controlled variable $x(k+1)$ in the next moment, and then further traverse all switching states based on this prediction to obtain the predicted value of the controlled variable $x(k+2)$ in moment $k+2$, which means the optimal switch is selected and applied to the system at moment $k+1$.

4. Results

To verify the effectiveness and feasibility of the proposed DQN-based current control method, a 3 × 3 DMC model is established, and the training of the DQN is handled with the use of the Reinforcement Learning Toolbox. Further, the experimental prototype (see Figure 4) has been built. The high-speed insulated gate bipolar transistor module (FF300R12KE4_E), which consists of two common-emitter-IGBTs, is used in the prototype. The controller includes a Digital Signal Processor (TMS320F28377) and Field Programmable Gate Array (10M50DAF484). The three-step commutation is implemented. The detailed model parameters are listed in

Table 1. Circuit parameters of the DMC.

Parameters	Value
Source phase voltage ($U_{\mathrm{s}}$)	50 V
Source voltage frequency ($f_{\mathrm{in}}$)	50 Hz
Sampling period ($T_{\mathrm{s}}$)	200 $\mathsf{\mu }$s
Input filter inductance ($L_{\mathrm{in}}$)	2 mH
Input filter capacitance ($C_{\mathrm{in}}$)	20 $\mathsf{\mu }$F
Input filter resistance ($R_{\mathrm{in}}$)	20 $\Omega$
Load frequency ($f_{\mathrm{o}}$)	70 Hz
Load resistance ($R_{\mathrm{o}}$)	10 $\Omega$
Load inductance ($L_{\mathrm{o}}$)	10 mH
Load current reference ($I_{\mathrm{o}\mathrm{m}}^{\ast }$)	3 A

, and the training parameters used in the DQN method are listed in

Table 2. Training parameters of the DQN method.

Parameters	Value
Discount factor ($\lambda$)	0.85
Hidden network layer number (l)	2
Hidden layer 1 neuron number ($n_1$)	6
Hidden layer 2 neuron number ($n_2$)	8
Target network update frequency ($N_T$)	20
Mini-batch size (M)	256
Replay buffer size (D)	$1\times {10}^5$
Maximum training steps (S)	1200
Maximum episode length (K)	2000

.

Figure 5 shows the output performance of the three-phase DMC with the MPC and proposed DQN methods. The input voltage of the DMC is set to 50 V, and a 3 A load current reference is imposed on the load. As is depicted in Figure 5, sinusoidal load currents are generated, which means the reference can be accurately tracked. From the perspective of waveform qualities, the proposed DQN method achieves a similar output performance to the MPC method.

$$\begin{array}{cc}\hfill \mathrm{MAE}& =\frac{1}{N}\sum _{k=1}^N|I\left(k\right)-I^{\ast }\left(k\right)|\hfill \\ \hfill \mathrm{MSE}& =\frac{1}{N}\sum _{k=1}^N{\left(I\left(k\right)-I^{\ast }\left(k\right)\right)}^2.\hfill \end{array}$$

(25)

To present the comparison of the two aforementioned control schemes clearly, some measurements (defined in Equation (15)) in the steady state are listed in

Table 3. System measurements of $i_{\mathrm{o}}$.

Control Method	Measurement	Value
MPC	THD ($i_{\mathrm{o}}$)	8.44%
	MAE ($i_{\mathrm{o}}$)	0.398
	MSE ($i_{\mathrm{o}}$)	0.202
DQN	THD ($i_{\mathrm{o}}$)	8.73%
	MAE ($i_{\mathrm{o}}$)	0.1536
	MSE ($i_{\mathrm{o}}$)	0.0396

. The values of the total harmonic distortion (THD) show that MPC achieves slightly better performance, but it has higher mean absolute errors (MAE) and mean square errors (MSE) due to the fact that MPC does not ensure a zero error in the steady state. Based on the results, it can be indicated that the proposed DQN method has almost the same performance as the MPC method.

The goal of the proposed DQN method is to train an agent that learns the best policy as it interacts with the environment so that, given any state, it will always take the most optimal action that produces the most reward in the long run. As for MPC, the best switching state is selected by solving an optimization problem at each time step. The objective is to minimize a cost function that captures the desired behavior and any penalties for violating constraints. In this paper, the agent is trained to learn the policy that is similar to MPC.

However, the proposed DQN method is not identical to the MPC method. First, the policy used in the proposed method is pre-trained, which alleviates the time-consuming traversal process in the MPC method. Second, in the training process, a discount factor is used to compute the expected reward, which not only helps the agent to learn more quickly but also ensures the future reward. In this sense, the DQN method is more like a multi-step MPC. Third, the RL-based method has the potential to take parameter variations, parasitic effects, and commutation processes into account through online training.

After the training process, the parameters of the learned agent policy are obtained by using the function “getLearnableParameters”. The weight w and bias b for each fully-connected layer are derived. The input variable x consists of the sampled input phase voltage, output load current, and the errors between the measured and the reference load current. A Relu function is adopted as the activation function for the output $y=wx+b$ of each hidden layer, which sets the negative value of y to zero. At last, the action corresponding to the output layer neuron with the maximum value is selected as the optimal switching state in this sampling interval.

Finally, experimental tests are conducted to verify the effectiveness of the proposed method. As shown in Figure 6, lower THD values are achieved by the MPC method. However, in the proposed DQN method, the agent is trained in a Simulink environment, which fails to consider the influence of the three-step commutation process of the DMC. Further, the training process of the agent might be improved, and the neural network can be optimized, which is beyond the scope of this paper. Although a deteriorated output current waveform is generated by the DQN method, the enumerating process in the MPC method is excluded for the reason that the agent is trained before its application. In this sense, the calculation time in each sampling period is significantly reduced, which means that the output performance of the proposed method can be improved by a higher sampling frequency.

5. Conclusions

In this paper, a novel DQN-based current control methodology for DMC systems was presented. By incorporating the DQN algorithm with the conventional current control method, we considered a fundamentally different solution to long-standing research problems with the use of an RL method. In addition, performance evaluations were provided to demonstrate the effectiveness and feasibility of the proposed methodology for DMCs. In the simulation, we showed that the proposed methodology can reduce the computational burden in comparison to the MPC method while maintaining feasible control performance. First, the time-consuming traversal process is replaced by an offline trained agent, making the proposed method available for a higher sampling frequency. Second, the agent is trained to ensure a zero error in the steady state, which achieves a smaller value of MAE and MSE in comparison to the MPC method. Third, the policy learned by the proposed method selects the optimal switching states in a similar manner to the MPC method. Therefore, the proposed DQN method achieves a similar output performance as the MPC method. However, in experiments, the proposed method fails to achieve a lower THD for the following reasons. First, the agent is trained in the Simulink environment, which neglects the commutation process and parasitic effects of the DMC. Second, the neural network can be improved by adding more hidden layers and neurons so as to fit the nonlinearity mapping from the high-dimensional input to the output. Finally, possible interesting directions for future research could be controlling multiple objectives such as common-mode voltage reduction and efficiency improvement and training an agent online.