Model-Free Deep Recurrent Q-Network Reinforcement Learning for Quantum Circuit Architectures Design

Abstract

Artificial intelligence (AI) technology leads to new insights into the manipulation of quantum systems in the Noisy Intermediate-Scale Quantum (NISQ) era. Classical agent-based artificial intelligence algorithms provide a framework for the design or control of quantum systems. Traditional reinforcement learning methods are designed for the Markov Decision Process (MDP) and, hence, have difficulty in dealing with partially observable or quantum observable decision processes. Due to the difficulty of building or inferring a model of a specified quantum system, a model-free-based control approach is more practical and feasible than its counterpart of a model-based approach. In this work, we apply a model-free deep recurrent Q-network (DRQN) reinforcement learning method for qubit-based quantum circuit architecture design problems. This paper is the first attempt to solve the quantum circuit design problem from the recurrent reinforcement learning algorithm, while using discrete policy. Simulation results suggest that our long short-term memory (LSTM)-based DRQN method is able to learn quantum circuits for entangled Bell–Greenberger–Horne–Zeilinger (Bell–GHZ) states. However, since we also observe unstable learning curves in experiments, suggesting that the DRQN could be a promising method for AI-based quantum circuit design application, more investigation on the stability issue would be required.

1. Introduction

Recent advances in artificial intelligence (AI) and Noisy Intermediate-Scale Quantum (NISQ) technology produce new perspectives in quantum artificial intelligence [1,2]. The control of quantum system by a classical agent has been studied in various settings [3,4,5]. Reinforcement learning (RL) [6,7,8,9,10,11,12] was successfully applied to control problems [11,13] of classical systems and fully observable Markov Decision Process (MDP) environments [14]. However, the control and learning of Partially Observable Markov Decision Process (POMDP) [15,16,17,18,19] is more difficult due to indirect access to the state information. Both planning [20] and learning [21] of POMDP are proposed. For a POMDP system, the underlying state transition is classical Markovian and is different from quantum dynamics. The quantum counterpart of POMDP, Quantum Observable Markov Decision Process (QOMDP) [22,23,24], was theoretically studied. Implementation of a QOMDP planning method for quantum circuits [2,25,26,27,28,29,30,31,32,33] is studied in a previous work [34]. Comparing to state tomography-based methods, which require an exponentially large number of measurement shots with respect to the circuit width, QOMDP-based approaches have favorable sample complexity from quantum circuits. However, an exact QOMDP planning method requires exponentially expensive classical computing. It is desirable to explore approximation methods to reduce the cost of computational resources.

Applying deep artificial neural networks for function approximations in reinforcement learning is known as deep reinforcement learning (DRL) [6]. DRL can be applied to quantum control [35,36,37,38,39,40,41,42,43,44,45,46]. Deep Q-network (DQN) [7,11] learning is a reinforcement learning method using deep artificial neural networks for the Q value function approximation. The traditional DQN method uses deep neural networks for the state-action Q-function for fully observable MDP. The deep recurrent Q-network (DRQN) method is proposed to encode the history sequence to tackle POMDP problems [47,48,49,50].

In this work, we implement a deep recurrent Q-learning agent for model-free reinforcement learning [47,48,49,50] to design quantum circuits. The DRQN is based on long short-term memory (LSTM) [51,52,53] networks that encode the action-observation history time-series for partially observable environments [49,50]. The fidelity achieved by the DRQN learning agent is improved over learning episodes, showing the effectiveness of the proposed algorithm. However, we also observe unstable learning curves in experiments. These observations suggest that the DRQN could be a promising method for AI-based quantum circuit design application, but more investigation on the stability issue would be required.

Many previous works for quantum control using different approaches can be found in the literature [35,36,37,38,39,40,41,42,43,44,45,46]. Borah et al. and Baum et al. [35,42] use a policy gradient. Niu et al. [37] use an on-policy method. He et al. [38] use a DQN. Bukov et al. [39] use a Q-table. Mackeprang et al. [40] use a DQN and double DQN. Zhang et al. [41] provide comparative study of Q-table, DQL, and policy gradient methods. August and Hernández-Lobato [46] use LSTM for the policy gradient. All these works [35,37,38,39,40,41,42,46] are controlled at the Hamiltonian level instead of at the circuit architecture level [34,43,44,45]. Kuo et al. and Pirhooshyaran and Terlaky [43,44] use a policy gradient. Ostaszewski et al. [45] use a double DQN. We note that Sivak et al.’s model-free paper [36] has several similarities and differences compared to our work. Sivak et al. applied an actor–critic policy gradient method to a quantum optical system with a continuous action space. Our work applied deep recurrent Q-learning to a qubit system with discrete action set. Both Sivak et al.’s method and our method are model-free and use LSTM. Sivak et al. use LSTM for the policy network and the value network over a continuous action space. We use LSTM for a history-dependent Q-function over a discretize action space, which is more practical for field application.

This work is organized as follows. Section 2 introduces the LSTM-based DRQN reinforcement learning method for quantum circuit architecture. Section 3 presents the simulation results. Section 4 provides some discussion. Section 5 is the conclusion.

2. Methods

2.1. MDP, POMDP, and QOMDP

A POMDP problem instance is described by a set of states $S$, a set of actions $A$, a set of observations $\mathsf{\Omega }$, a state transition probability $P$, an observation probability $O$, a reward function $R$, and a discount rate $\gamma \in \left[\mathbf{0},\mathbf{1}\right]$. At each time step $t$, the agent in state $s_t\in S$ takes an action $a_t\in A$ and moves to a new state $s_{t+1}~P(s^{\prime }|s_t,a_t)$. The agent also receives an observation $o_t~O(o|s_t)$, $o_t\in \mathsf{\Omega }$ and a reward $r_t=R(s_t,a_t,s_{t+1})\in ℝ$. The action-observation history time series is ${𝒽}_t=\{a_1,o_1,a_2,o_2,\dots a_t,o_t\}$. The goal is to find a policy $\pi (a|h)$ to optimize the expected future reward ${\mathbb{E}}_{\pi }\left[{\displaystyle \sum }_{i=t}^T{\gamma }^{i-t}{r}_i\right]$. In contrast to the situation of MDP, a POMDP agent does not have access to the time series $\{s_t\}$.

A QOMDP problem instance is described by a Hilbert space $\mathcal{S}$, a set of action super-operators $\mathcal{A}$, a set of observations $\mathsf{\Omega }$, a set of reward operators $ℛ$, a discount rate $\gamma \in \left[0,1\right]$, and an initial quantum state $|s_0\rangle$. The set of actions consists of super-operators $\mathcal{A}=\{A^{a^1},\dots ,A^{a^{|\mathcal{A}|}}\}$, where each super-operator $A^a=\{A_{o^1}^a,\dots ,A_{o^{|\mathcal{O}|}}^a\}$ has $|\mathcal{O}|$ Kraus matrices. At each time step $t$, the agent takes an action $a_t$, which introduces a change of the state of current quantum system

$$|s_t\rangle ⟼\frac{A_{o_t}^{a_t}|s_t\rangle }{\sqrt{\langle s_t|A_{o_t}^{a_t†}{A}_{o_t}^{a_t}|s_t\rangle }}$$

The agent also receives an observation $o_t~\mathrm{Pr}(o||s_t\rangle ,a_t)=\langle s_t|A_o^{a_t†}{A}_o^{a_t}|s_t\rangle$, $o_t\in \mathsf{\Omega }$ and a reward $r_t=\langle s_t|R_{a_t}|s_t\rangle \in ℝ$, where $R_{a_t}\in ℛ$. Similar to POMDP and MDP, the goal is to find a policy to optimize the expected future reward.

2.2. LSTM-Based Deep Recurrent Q-Network

LSTM is a type recurrent neural network which can be used to model sequential data. The hidden state $h_t$ and output $c_t$ are computed by the recurrence $(h_t,c_t)=LSTM(h_{t-1},c_{t-1},x_{t-1})$ for time-dependent input signal $x_t$. Traditional Q-learning for observable MDP uses a state-action Q-function $Q(s_t,a_t)$ to represent the value of an action $a_t$ at a known state $s_t$. To deal with partially observable environments in which $s_t$ is unknown, a history-dependent Q-function $Q(a_t,{𝒽}_{t-1})$ is used instead of the state-action Q-function. By treating the action-observation pair as input $x_t=(a_t,o_t)$, LSTM enables the encoding of the history-dependent Q-function $Q(a_t,{𝒽}_{t-1})$. A feed-forward neural network (FNN) is concatenated with the LSTM output to represent the Q-function. The FNN is a simple linear transformation, and its output gives the Q-value $Q(:,{𝒽}_{t-1})=W{c}_{t-1}+b$, where $W\in {ℝ}^{|\mathcal{A}|\times |h|}$ is a trainable weight matrix, and $b\in {ℝ}^{|h|}$ is a bias vector. $|h|$ denotes the size of the LSTM hidden states. The LSTM–FNN structure is shown in Figure 1a. The update of the Q function is via the optimization of loss function

$$L={(Q(a_t,{𝒽}_{t-1})-(r_{t-1}+\gamma \underset{A}{\max }Q(A,{𝒽}_t)))}^2$$

which can be computed by back-propagation through time. The implementation is performed by using the package PyTorch [54]. The hyperparameters can be found in

Table 1. List of hyperparameters.

Hyperparameter	Value
Target state fidelity threshold	0.99
Maximum steps per episode	100
Number of episodes	30,000
Reply buffer size	1,000,000
Epsilon start	1.0
Epsilon end	0.01
Epsilon decay rate	0.9997
LSTM sequence length	3
LSTM hidden states size	30
FNN hidden states size	30
FNN activation function	linear
Minibatch size	32
Learning rate	0.001
Soft update rate tau	0.001
Discount rate	0.95

.

2.3. RL Method

The proposed method is depicted in Figure 1b. The RL environment is the quantum circuit to be designed. The classical agent receives 0–1 observation from measurement result of the ancillary qubit. The action–observation pair is used to update the DRQN, and then the decision for the next action is made by the agent to control the circuit. The reward is the fidelity with respect to the target state $r_t=\langle s_t|s_{target}\rangle \langle s_{target}|s_t\rangle$. The policy is epsilon-greedy. Experience reply is used to stabilize the calculation. Using the convention that the Hilbert space is $ancilla\otimes system$, and the operator in Figure 1b is $U(a_t)=U_{ent}(H\otimes U_{action})$, where $H$ is the single qubit Hadamard gate acting on the ancillary qubit. The action unitary $U_{action}$ is chosen from the action set $\{CNO{T}_{i,j}:i,j\in system\}{\displaystyle \cup }\{R_{d,i}(\theta ):i\in system,\theta \in \{\pm \frac{\pi }{9}\},d\in \{X,Y,Z\}\}$. Here, $CNO{T}_{i,j}$ denotes the control-not gate, for which the i-th qubit is the control qubit and the j-th qubit is the target qubit. $R_{d,i}(\theta )$ denotes single qubit rotation of i-th qubit around d-axis. The system–ancilla entangler $U_{ent}={\displaystyle \prod }_{i\in system}CNO{T}_{i,ancilla}$ computes the system parity function and outputs the result to an ancilla qubit. The setup is similar to that of [34], but the classical agent in this work is an RL agent instead of a planning agent.

3. Results

Numerical simulations are conducted to test the applicability of the proposed method. The simulation code is based on the packages Numpy [55], Matplotlib [56], PyTorch [54], and Qiskit [57]. We test the state generation task for the 2-qubit Bell state and 3-qubit Greenberger–Horne–Zeilinger (GHZ) state [58]. The target state is considered reached when the fidelity is larger than a threshold value 0.99. The maximum number of steps for each episode is set to be 100. The PyTorch hyperparameters are listed in Table 1.

Figure 2 is the learning curves for the 2-qubit Bell state. The received reward and number of steps to reach the target state are plotted with respect to the number of learning episodes. Each curve is the moving average of 2000 episodes and 10 independent runs. The error bar denotes the one standard deviation over 10 independent runs. For 30,000 episodes, we observe that the average reward is increased from <0.3 to >0.4. The maximum of the one-sigma error bar is close to 0.65. The average number of steps to reach the goal is decreased from >95 to <90. The minimum of the one-sigma error bar is close to 60.

Figure 3 shows the learning curves for 3-qubit GHZ state. For 30,000 episodes, we observe that the reward is increased from <0.15 to >0.3. The maximum of one-sigma error bar can be larger than 0.45. The average number of steps to the goal is larger than 99 throughout the learning episodes.

Figure 4 is the city diagram for the density matrix generated by the RL agent. The result is the highest fidelity result over 10 independent training runs and 100 test steps for each training obtained by the policy of the last (30,000th) training episode. The fidelity of the obtained density matrix is 0.9698 for the Bell state, and the fidelity of the obtained density matrix is 0.6710 for the GHZ state.

4. Discussion

From the experimental data in Figure 2 and Figure 3, we observe that the fidelity of the 2-qubit Bell state and 3-qubit GHZ state are improved by the proposed learning algorithm. However, since these values are mostly way below the stopping criteria 0.99, the number of steps is not improved significantly. The best output state has high fidelity with respect to the target for the 2-qubit case, while the 3-qubit case provides moderate fidelity. These results demonstrate that the learning algorithm is effective, but the performance within our experiments is not satisfactory. More learning episodes and fine-tuning of hyperparameters could potentially improve the performance. The fidelity achieved in the 2-qubit Bell experiments is generally better than that of the 3-qubit GHZ experiments. This is reasonable, since the possible action space for the 2-qubit system is smaller, and the required action sequence to produce a 2-qubit Bell state is shorter than that of a 3-qubit GHZ state.

The city diagram in Figure 4 allows us to visualize the states produced by the agent. The Bell–GHZ target state is $\frac{1}{\sqrt{2}}(|00+|11\rangle )$ for two qubits and $\frac{1}{\sqrt{2}}(|000+|111\rangle )$ for three qubits. The ideal city diagram has peaks at four corners of the real part. For the 2-qubit case, the experimental data resemble the ideal case, and, hence, the fidelity is higher. On the other hand, the 3-qubit city diagram has many sub-peaks, which implies low fidelity.

To further understand the reasons behind the limitation of our method, the test fidelity distribution histogram for 10 independent runs is plotted in Figure 5. It is observed that all samples lie in the region $Fidelity>0.4$ for both the 2-qubit and 3-qubit cases. However, the 2-qubit result has the highest fidelity sample in the interval $Fidelity\in [0.9,1.0)$, while the 3-qubit result has the highest fidelity sample in the interval $Fidelity\in [0.6,0.7)$. The 2-qubit result not only has better best-case performance but also has distribution maximum at $Fidelity\in [0.6,0.7)$. This is better than the peak location of the 3-qbuit result, which is $Fidelity\in [0.4,0.5)$. The problem is that a learning method that is successful for small problem instances would not necessarily scale to larger problem instances. We are encountering an scalability issue that arises commonly in the application of machine learning methodologies to optimization problems [59]. To the best of our knowledge, this is still an unresolved issue in the community, so further investigation in this direction is desirable.

5. Conclusions

In this work, we propose and implement a deep recurrent Q-network algorithm for quantum circuit design. Experimental results show that the agent is able to learn to produce a better quantum circuit for entangled states’ preparation. However, the learned fidelity is not satisfactory. Future research and development are required to improve the quality of the state-generation task. In particular, scalability to larger problem instances should be tackled. It would also be desirable to explore other applications, for example, the energy minimization task [26,34,60,61,62].