Scaling Up Q-Learning via Exploiting State–Action Equivalence

Abstract

Recent success stories in reinforcement learning have demonstrated that leveraging structural properties of the underlying environment is key in devising viable methods capable of solving complex tasks. We study off-policy learning in discounted reinforcement learning, where some equivalence relation in the environment exists. We introduce a new model-free algorithm, called QL-ES (Q-learning with equivalence structure), which is a variant of (asynchronous) Q-learning tailored to exploit the equivalence structure in the MDP. We report a non-asymptotic PAC-type sample complexity bound for QL-ES, thereby establishing its sample efficiency. This bound also allows us to quantify the superiority of QL-ES over Q-learning analytically, which shows that the theoretical gain in some domains can be massive. We report extensive numerical experiments demonstrating that QL-ES converges significantly faster than (structure-oblivious) Q-learning empirically. They imply that the empirical performance gain obtained by exploiting the equivalence structure could be massive, even in simple domains. To the best of our knowledge, QL-ES is the first provably efficient model-free algorithm to exploit the equivalence structure in finite MDPs.

1. Introduction

Reinforcement learning (RL) aims to develop computer systems with the ability to learn how to behave optimally, or nearly so, in an unknown dynamic environment. An RL task typically involves an agent interacting with the environment, which is often modeled as a Markov decision process (MDP), and the agent’s goal is to find a policy maximizing some notion of reward. In most settings in RL, the MDP is initially unknown beyond its state and action spaces. Hence, the agent aims to learn a near-optimal policy using the experiences collected from the environment.

A classical setting in RL is off-policy learning [1], where one tries to learn the optimal action–value function (i.e., Q-function) through the data collected under some behavior or logging policy. Perhaps the most famous off-policy learning algorithm is the celebrated Q-learning algorithm [2], whose improved variants, combined with deep neural networks as function approximators, played key roles in many recent breakthroughs in RL [3,4]. More precisely, Q-learning and its variants fall under the category of model-free methods, in which one tries to directly estimate the optimal value function—without estimating the true model (MDP)—from the collected experience, from which a near-optimal policy could be straightforwardly derived. This approach is in stark contrast to the model-based counterpart, where one first attempts to estimate the unknown model parameters (i.e., MDP parameters that include transition probabilities and rewards) from the collected experience and then finds an optimal policy in the estimated model.

Off-policy learning in finite MDPs is by now well understood, and the existing literature (e.g., [5,6,7,8,9]) exhibit algorithms that admit PAC-type sample complexity guarantees. Precisely speaking, these algorithms are guaranteed to return a near-optimal policy, with respect to a prescribed accuracy, with high probability if the amount of collected experience exceeds a certain (algorithm-dependent) function of the MDP parameters (and relevant input parameters). Most of these works study unstructured tabular MDPs, where the advertised sample complexity bounds scale, among other things, with the size of the state–action space. Thus, despite their appealing performance guarantees, most of these algorithms only work reasonably well when the size of the underlying MDP is small. On the other hand, many practical tasks can be modeled by MDPs with huge state spaces (or even infinite), but they often exhibit some structural properties. Ignoring such structural properties and directly applying the above algorithms would lead to a prohibitively large sample complexity bound, which may imply a huge learning phase in the worst case. Alternatively, one could leverage the structure in the MDP to speed up the exploration. In fact, exploiting the structure allows the agent to use the collected observations from the environment to reduce the uncertainty in the model parameter for many similar state–action pairs at each time slot. As a result, the learning performance would depend on the effective size of the state space (or the effective number of unknown parameters). Various notions of structures have been studied in MDPs, which include the Lipschitz continuity of MDP parameters (e.g., rewards and transition functions) [10,11,12,13], factorization structure [14,15,16], and equivalence relations [17,18,19,20,21,22]. These works reveal that exploiting the underlying structure in the environment in various RL tasks leads to massive empirical performance gain (over structure-oblivious algorithms) and to significantly improved performance bounds. However, exploiting structure often poses additional challenges.

This work is motivated by tabular RL problems, where the (potentially large) state–action space admits a natural partitioning such that within each element of the partition (or class), the state–action pairs have similar transition probabilities. There exist several ways to characterize the similarity between the transition distribution of two state–action pairs. Here, we consider a notion implying that they are (almost) identical up to some permutation. As we shall see in later sections, this notion of structure induces an equivalence relation in the state–action space. This model has been considered in prior work [18,23,24], where model-based algorithms were presented to exploit such a structure in the context of regret minimization in episodic or average-reward MDPs. However, their proposed ideas and techniques are specific to a model-based approach, where the model parameters are directly estimated, and cannot be used to incorporate the knowledge of the equivalence structure into a model-free algorithm. Model-free algorithms have played a pivotal role in the recent success of RL to solving complex tasks arising in real-world applications (e.g., autonomous driving and continuous control [25]). Hence, it sounds promising to study the gain one could obtain using model-free methods when leveraging the equivalence structure, thereby extending the theoretical analysis in [18,24] beyond model-based methods.

Contributions. We make the following contributions. We study off-policy learning in discounted finite MDPs, admitting some equivalence structure in their state–action space. We introduce a new model-free algorithm, called QL-ES (Q-learning with equivalence structure), which is a variant of (asynchronous) Q-learning tailored to exploit the equivalence structure in the MDP, when a prior knowledge on the structure is provided to the agent. We report a non-asymptotic PAC-type sample complexity bound for QL-ES, thereby establishing its sample efficiency. This bound also allows us to quantify the superiority of QL-ES over Q-learning analytically. As it turns out, the sample efficiency gain of QL-ES over Q-learning is captured by an MDP-dependent quantity $\xi$ that is defined in terms of the associated covering times in the MDP; see Section 5 for details. Analytically establishing the dependence of the gain ratio $\xi$ on the number S of states in a given MDP seems difficult, although it is possible to numerically compute it. Nonetheless, we present a simple example where $\xi =O\left(S\right)$, thus showcasing that QL-ES in some domains may require much fewer (by a factor of S) samples than Q-learning. Furthermore, we numerically compute $\xi$ for a few families of MDPs built using standard environments (with increasing S), thereby showcasing the theoretical superiority of QL-ES over Q-learning. Through extensive numerical experiments on standard domains, we show that Q-function estimates under QL-ES converge much faster than those obtained from (structure-oblivious) Q-learning. These results demonstrate that the empirical performance gain from exploiting the equivalence structure could be massive, even in simple domains. To our best knowledge, QL-ES is the first provably efficient model-free algorithm to exploit the equivalence structure in MDPs.

2. Related Work

Similarity and equivalence in MDPs. There is a rich literature on learning and exploiting various notions of structure in MDPs, where the aim is to leverage structure to alleviate the computational cost of finding an optimal policy (in the known MDP setting) or to speed up exploration (in the RL setting). Many such algorithms fall into the category of state abstraction (or aggregation) [26,27]. Approximate homomorphism is proposed to construct beneficial abstract models in MDPs [28]. In the known MDP setting, Refs. [29,30] appear to be the first presenting the notion of equivalence between states based on stochastic bi-simulation. The authors of [31,32] use bi-simulation metrics as quantitative analogues of the equivalence relations to partition the state space by capturing similarities. In the RL setting, Refs. [18,19,20,33,34] investigate model-based algorithms that rely on the grouping of similar states (or state–action pairs) to speed up exploration. Ref. [20] is the first to present an average-reward RL algorithm (in the regret setting), where the confidence intervals of similar states are aggregated. Ref. [18] studies regret minimization in average-reward MDPs with equivalence structure and presents the C-UCRL algorithm, which is capable of exploiting the structure. The regret bound for C-UCRL depends on the number of classes in the MDP rather than the size of the state–action space. A similar equivalence structure was studied in [17] in the context of multi-task RL, where similarities of the transition dynamics across tasks were extracted and exploited to speed up learning. Ref. [24] studies the efficiency of hierarchical RL in the regret setting in scenarios where the hierarchical structure is defined with respect to the notion of equivalence; more precisely, it assumes that the underlying MDP can be decomposed into equivalent sub-MDPs—i.e., smaller MDPs with identical reward and transition functions up to some known bijection mappings. Closest to our work, in terms of the structure definition, is [18]. However, we restrict ourselves to a model-free approach where the model-based machinery presented in [18] does not apply. Finally, we mention that there is some literature on exploiting equivalence in deep RL (e.g., [21,22]). However, none of these works study provably efficient learning methods to our best knowledge.

Q-learning and its variants. We provide a very brief overview of the works studying theoretical analysis of Q-learning and its variants. Q-learning [2] has been around for more than three decades as a cheap and popular model-free method to solve finite, unknown discounted MDPs without estimating the model. Its convergence was investigated in an asymptotic flavor [35,36], and more recently in the non-asymptotic (finite-sample) regime in a series of work, including [9,37,38,39,40]. To the best of our knowledge, Ref. [9] reports the sharpest PAC-type sample complexity bound for the classical Q-learning. Some of these works present variants of Q-learning with improved sample complexity bounds using a variety of techniques, such as acceleration and variance reduction [9,40,41]. Although the concept of equivalence in MDPs is not new, there is no work reporting PAC-type sample complexity bounds for model-free algorithms combined with equivalence relations, to our knowledge.

3. Problem Formulation

In this section, we present some necessary background and formulate the reinforcement learning problem considered in this paper. We use the following notations throughout. For a set B, $\Delta \left(B\right)$ denotes the set of all probability distributions over B. For an event E, $\mathbb{I}E$ denotes the indicator function of E: namely, it equals 1 if E holds, and 0 otherwise.

3.1. Discounted Markov Decision Processes

Let $M=(\mathcal{S},\mathcal{A},P,R,\gamma )$ be an infinite-horizon discounted MDP [42], where $\mathcal{S}$ denotes a discrete state space with cardinality S, $\mathcal{A}$ denotes a discrete action space with cardinality A, and $\gamma \in (0,1)$ is a discount factor. $P:\mathcal{S}\times \mathcal{A}\to \Delta \left(\mathcal{S}\right)$ represents the transition function such that $P\left(s^{\prime }\right|s,a)$ denotes the probability of transiting to state $s^{\prime }$ when action $a\in \mathcal{A}$ is chosen in state $s\in \mathcal{S}$. Further, $R:\mathcal{S}\times \mathcal{A}\to [0,1]$ denotes the reward function supported on $[0,1]$ such that $R(s,a)$ denotes the reward distribution when choosing action $a\in \mathcal{A}$ in state $s\in \mathcal{S}$. We denote by $\overline{R}(s,a)$ the mean of $R(s,a)$. A stochastic (or randomized) policy $\pi :\mathcal{S}\to \Delta \left(\mathcal{A}\right)$ is a mapping that maps a state to a probability distribution over $\mathcal{A}$. For a policy $\pi$, the value function of $\pi$ is a mapping $V^{\pi }:\mathcal{S}\to \mathbb{R}$ defined as

$$V^{\pi }\left(s\right):=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t|s_0=s\right],s\in \mathcal{S},$$

where for all $t\ge 0$, $a_t\sim \pi \left(s_t\right)$, $s_{t+1}\sim P(·|s_t,a_t)$, and $r_t\sim R(s_t,a_t)$, and where the expectation is taken with respect to the randomness in rewards, next states, and actions sampled from $\pi$. The action–value function of a policy $\pi$, denoted by $Q^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, is defined as

$$Q^{\pi }(s,a):=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t|s_0=s,a_0=a\right],s\in \mathcal{S},a\in \mathcal{A}.$$

The optimal value function is denoted by $V^{\star }$ and satisfies $V^{\star }\left(s\right)={\max }_{\pi }{V}^{\pi }\left(s\right)$ for all $s\in \mathcal{S}$. It is well known that in any finite MDP, there exists a stationary deterministic policy ${\pi }^{\star }:\mathcal{S}\to \mathcal{A}$ such that $V^{{\pi }^{\star }}=V^{\star }$, which is called an optimal policy [42]. Similarly, the optimal state–action value function is defined as $Q^{\star }(s,a)={max}_{\pi }{Q}^{\pi }(s,a)$ for all $(s,a)\in \mathcal{S}\times \mathcal{A}$. An optimal policy ${\pi }^{\star }$ satisfies ${\pi }^{\star }\left(s\right)\in arg{max}_a{Q}^{\star }(s,a)$ for all $s\in \mathcal{S}$. Furthermore, $Q^{\star }$ is the unique solution to the optimal Bellman equation [42]:

$$Q^{\star }(s,a)=\overline{R}(s,a)+\gamma \sum _{s^{\prime }\in \mathcal{S}}P\left(s^{\prime }\right|s,a)\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{\star }(s^{\prime },a^{\prime }),s\in \mathcal{S},a\in \mathcal{A}.$$

3.2. The Off-Policy Learning Problem and Q-Learning

We consider the off-policy learning problem as follows. The agent is provided with some dataset $\mathcal{D}$ collected according to some behavior or logging policy ${\pi }_{\mathrm{b}}$. Precisely speaking, $\mathcal{D}$ takes the form of trajectory ${\left\{(s_t,a_t,r_t)\right\}}_{t\ge 0}$, where $s_0$ is some initial state, and where for each $t\ge 0$, $a_t\sim {\pi }_{\mathrm{b}}\left(s_t\right)$, $s_{t+1}\sim P(·|s_t,a_t)$, and $r_t\sim R(s_t,a_t)$. The agent is given an accuracy parameter $\epsilon >0$ and a failure probability parameter $\delta \in (0,1)$, and its goal is to find an $\epsilon$-optimal policy using as few samples as possible from $\mathcal{D}$.

We need to impose some assumptions on the behavior policy ${\pi }_{\mathrm{b}}$ to ensure that it is possible to learn a near-optimal policy only using $\mathcal{D}$ efficiently with PAC-type guarantees. To state the assumptions, we introduce some necessary definitions, which are borrowed from standard textbooks on Markov chains (e.g., [43]) but are also standard in the theoretical analysis of Q-learning (e.g., [9,39]). Let $\mathcal{X}$ be a finite set. The total variation distance between two distributions $\mu$ and $\nu$ defined over $\mathcal{X}$ is $d_{\mathrm{TV}}(\mu ,\nu ):=\frac{1}{2}{\sum }_{x\in \mathcal{X}}\left|\mu \left(x\right)-\nu \left(x\right)\right|$. Now, consider an ergodic Markov chain ${\left(X_t\right)}_{t\ge 1}$ with state space $\mathcal{X}$ and transition function $p\in \Delta \left(\mathcal{X}\right)$, and let $\mu$ be the unique stationary distribution of the chain. The Markov chain is said to be uniformly ergodic if there exist some $\rho <1$ and $M<\infty$ such that for all $t>0$,

$$\underset{x\in \mathcal{X}}{sup}{d}_{\mathrm{TV}}(\mu ,p^t(·|x))\le M{\rho }^t,$$

where $p^t(·|x)$ is the distribution of $X_t$ given $X_0=x$.

Similar to [9], we assume that the Markov chain induced by ${\pi }_{\mathrm{b}}$ is uniformly ergodic. This property ensures that all the states are visited infinitely often, and that convergence to the stationary distribution is performed at a geometric pace. This property is needed for the result presented in Section 5.

The Q-learning algorithm. The Q-learning algorithm [35] is perhaps the most famous model-free algorithm for learning an optimal policy in unknown tabular MDPs. As a model-free method, it directly learns the optimal Q-function $Q^{\star }$ of the MDP (without estimating P and R), which can be used to derive a policy. The algorithm maintains an estimate $Q_t$ of the optimal $Q^{\star }$ at each time step t. Specifically, it starts from an arbitrary choice for $Q_0\in {\mathbb{R}}^{S\times A}$ and updates $Q_t$, at each $t\ge 0$, as

$$\begin{array}{c}\hfill Q_{t+1}(s_t,a_t)=\left\{\begin{array}{cc}(1-{\alpha }_t)Q_t(s,a)+{\alpha }_t\left(r_t+\gamma \underset{a^{\prime }\in \mathcal{A}}{max}{Q}_t(s_{t+1},a^{\prime })\right),\hfill & (s,a)=(s_t,a_t),\hfill \\ Q_{t+1}(s,a)=Q_t(s,a),\hfill & (s,a)\ne (s_t,a_t),\hfill \end{array}\right.\end{array}$$

(1)

where ${\alpha }_t$ is a suitably chosen learning rate. Precisely speaking, the update Equation (1) corresponds to the asynchronous variant of Q-learning. The classical asymptotic performance analysis of Q-learning (in, for example, [36]) indicates that if (i) ${\pi }_{\mathrm{b}}$ is exploratory enough such that all state–action pairs are visited infinitely often and (ii) ${\left({\alpha }_t\right)}_{t\ge 0}$ satisfies the following conditions, known as the Robbins–Monro conditions [36,44]:

$${\alpha }_t\ge 0,\sum _{t=0}^{\infty }{\alpha }_t=\infty ,\sum _{t=0}^{\infty }{\alpha }_t^2<\infty ,$$

then $Q_t{\to }_{t\to \infty }{Q}^{\star }$ almost surely, for any choice of $Q_0\in {\mathbb{R}}^{S\times A}$. For example, one such choice of the learning rate is as follows: for all $t\ge 0$, ${\alpha }_t=\frac{1}{N_t(s_t,a_t)+1}$, where for any $(s,a)$, $N_t(s,a)$ denotes the number of visits to $(s,a)$ up to time t: $N_t(s,a)={\sum }_{\tau =0}^{t-1}\mathbb{I}\{(s_{\tau },a_{\tau })=(s,a)\}$. The pseudo-code of Q-learning is described in Algorithm 1, where the learning rate sequence ${\left({\alpha }_t\right)}_{t\ge 0}$ is considered as input.

Algorithm 1 Q-learning [2].

Input: dataset $\mathcal{D}$, maximum iterations T, learning rates ${\left({\alpha }_t\right)}_{t\ge 0}$

Initialization:  $Q_0=0\in {\mathbb{R}}^{S\times A}$

for  $t=0,1,\dots ,T$ do

Sample action $a_t\sim {\pi }_{\mathrm{b}}\left(s_t\right)$ and observe $r_t\sim R(s_t,a_t)$ and $s_{t+1}\sim P(·|s_t,a_t)$.

Compute $Q_{t+1}$ using (1).

end for

It is worth remarking that some studies consider off-policy learning in the online setting, where data are collected from the environment while executing the algorithm. In such online settings, it is possible to choose $a_t$ according to an adaptive (randomized) policy ${\pi }_t$ (usually defined as a function of the current estimate $Q_t$), instead of sampling it from a fixed behavior policy. In doing so, the aim is to balance exploration and exploitation so as to collect higher rewards while learning the Q-function. A notable example is the $ϵ$-greedy policy, where at time t, $a_t$ is chosen greedily with respect to $Q_t(s_t,·)$ with probability $1-ϵ$, and chosen uniformly at random (from $\mathcal{A}$) with probability $ϵ$. In the theoretical part of this paper, we consider a fixed behavior policy.

3.3. Similarity and Equivalence Classes

We now present a definition of the equivalence structure considered in this paper. We start by stating the following definition of similarity in finite MDPs as introduced in [18]. A similar definition is provided in [23].

Definition 1

(Similar state–action pairs [18]). Two state–action pairs $(s,a)$ and $(s^{\prime },a^{\prime })$ are called $\theta$-similar if there exist mappings ${\sigma }_{s,a}:\{1,\dots ,S\}\to \mathcal{S}$ and ${\sigma }_{s^{\prime },a^{\prime }}:\{1,\dots ,S\}\to \mathcal{S}$ such that

$$\begin{array}{c}\hfill \parallel P\left({\sigma }_{s,a}(·)\right|s,a)-P\left({\sigma }_{s^{\prime },a^{\prime }}(·)\right|s^{\prime },a^{\prime }){\parallel }_1\le \theta .\end{array}$$

We refer to ${\sigma }_{s,a}$ as the  profile mapping  (or for short,  profile) for $(s,a)$, and denote by $\sigma ={\left({\sigma }_{s,a}\right)}_{s,a}$ the set of profile mappings across $\mathcal{S}\times \mathcal{A}$.

We stress that ${\sigma }_{s,a}$ in Definition 1 may not be unique in general. The case of 0-similarity is of particular interest: it is evident from Definition 1 that if $(s,a)$ and $(s^{\prime },a^{\prime })$ are 0-similar, then $P(·|s,a)$ and $P(·|s^{\prime },a^{\prime })$ are identical up to some permutation from $\mathcal{S}\times \mathcal{A}$ to $\mathcal{S}\times \mathcal{A}$. Furthermore, 0-similarity induces a partition of the state–action space $\mathcal{S}\times \mathcal{A}$ as formalized below.

Definition 2

(Equivalence structure [18]). 0-similarity is an equivalence relation and induces a canonical partition of $\mathcal{S}\times \mathcal{A}$. We refer to such a canonical partition as equivalence structure and denote it by $\mathcal{C}$. We further define $C:=\left|\mathcal{C}\right|$.

We provide an example to help understand Definition 2. Consider the RiverSwim environment [45] with 6 states and $\mathcal{A}=\{\mathrm{L},\mathrm{R}\}$ (see Figure 1). The two pairs $(s_1,\mathrm{R})$ and $(s_6,\mathrm{R})$ are 0-similar since $P(·|s_1,\mathrm{R})=[0.6,0.4,0,0,0,0]$ and $P(·|s_6,\mathrm{R})=[0,0,0,0,0.6,0.4]$, so there exist permutations ${\sigma }_{s_1,\mathrm{R}}$ and ${\sigma }_{s_6,\mathrm{R}}$ such that $P\left({\sigma }_{s_1,\mathrm{R}}(·)\right|s_1,\mathrm{R})=P\left({\sigma }_{s_6,\mathrm{R}}(·)\right|s_6,\mathrm{R})$. Additionally, all pairs $(s_i,\mathrm{L}),i=1,\dots ,6$ are 0-similar, and so are $(s_i,\mathrm{R}),i=2,\dots ,5$. We thus identify an equivalence structure $\mathcal{C}$ of $\mathcal{S}\times \mathcal{A}$ as follows: $\mathcal{C}=\{c_1,c_2,c_3\}$ with $c_1=\{(s_1,\mathrm{R}),(s_6,\mathrm{R})\}$, $c_2=\{(s_i,\mathrm{R}),i=2,\dots ,5\}$, and $c_3=\{(s_i,\mathrm{L}),i=1,\dots ,6\}$.

Note that for any finite MDP, Definition 2 trivially holds with $\mathcal{C}=\mathcal{S}\times \mathcal{A}$. There are many interesting environments that non-trivially admit the notion of equivalence structure in Definition 2. In such MDPs, it is often the case that the size C of the structure is much smaller than the size of the state–action space, i.e., $C\ll SA$. For example, in a generic RiverSwim with S states, one has $C=3$. Another example admitting an equivalence structure is the classical grid-world MDP, which is detailed in Section 6.2.

Off-policy learning in MDPs with equivalence structures. In this work, we assume that the underlying MDP M admits an equivalence structure $\mathcal{C}$ as introduced above. In other words, the transition function P is such that $\mathcal{S}\times \mathcal{A}$ can be partitioned into $C:=\left|\mathcal{C}\right|$ classes, where the pairs in each class $c\in \mathcal{C}$ are 0-similar. We make the following assumption regarding the agent’s prior knowledge about $\mathcal{C}$.

Assumption A1.

The agent has prior knowledge on $\mathcal{C}$.

Let $c(s,a)$ denote the class that the pair $(s,a)$ belongs to. Assumption A1 implies that the agent knows $c(s,a)$ for any pair $(s,a)$ and the associated profile mapping ${\sigma }_{s,a}$. Note, however, that the agent does not know the actual transition probabilities. Armed with such prior knowledge, we are interested in devising a model-free algorithm that is capable of leveraging the structure in M to improve the learning performance. We expect that the corresponding speed-up in learning the optimal Q-function could be significant in MDPs with $C\ll SA$.

We also make the following assumption regarding the reward function to ease the presentation. (This assumption has often been made in the literature on theoretical RL (e.g., [6,46]) since the main challenge in RL arises from unknown transition probabilities.)

Assumption A2.

The agent knows the reward function R.

4. The QL-ES Algorithm

In this section, we present a variant of Q-learning that exploits the equivalence structure in the environment to speed up the learning of the optimal Q-function. We call this algorithm QL-ES, which is short for ‘Q-learning with equivalence structure’.

QL-ES follows the same machinery of QL but is also built on the idea that the knowledge on $\mathcal{C}$ and the corresponding profile mappings allows for using the triplet $(s_t,a_t,s_{t+1})$ collected at each time t to update potentially multiple entries of $Q_t$. Precisely speaking, the Q-function update for a given pair $(s,a)$ requires a sample from $R(s,a)$ and a sample from $P(·|s,a)$. Since the agent perfectly knows $\mathcal{C}$, it can determine $c(s_t,a_t)$, i.e., the class $(s_t,a_t)$ belongs to. Hence, it knows all other pairs belonging to the same class as $(s_t,a_t)$. Then using $s_{t+1}$ (sampled from $P(·|s_t,a_t)$) at time t, the agent can construct samples for all other pairs in $c(s_t,a_t)$ as follows. If $(s,a)\in c(s_t,a_t)$, then there is a mapping ${\sigma }_{s,a}$ and a state $s_{t+1}^{\left(sa\right)}$ such that $P\left({\sigma }_{s_t,a_t}\left(s_{t+1}\right)\right|s_t,a_t)=P\left({\sigma }_{s,a}\left(s_{t+1}^{\left(sa\right)}\right)|s,a\right)$. In other words, the sample $s_{t+1}$ obtained from $P(·|s_t,a_t)$ is equivalent to obtaining a fresh sample $s_{t+1}^{\left(sa\right)}$ from $P(·|s,a)$. As ${\sigma }_{s,a}$ and ${\sigma }_{s_t,a_t}$ are known, the agent finds $s_{t+1}^{\left(sa\right)}:={\sigma }_{s,a}^{-1}\left({\sigma }_{s_t,a_t}\left(s_{t+1}\right)\right)$, where ${\sigma }^{-1}$ denotes the inverse mapping of $\sigma$. In other words, $s_{t+1}^{\left(sa\right)}$ acts as a counterfactual next-state for $(s,a)\in c(s_t,a_t)$ thanks to the knowledge on $\mathcal{C}$.

The agent thus uses $(s_t,a_t,s_{t+1})$ to update $Q_t$ for all $(s,a)\in c(s_t,a_t)$. In summary, we update $Q_t$ as follows: For all $t\ge 0$,

$$\begin{array}{c}\hfill Q_{t+1}(s_t,a_t)=\left\{\begin{array}{cc}(1-{\alpha }_t)Q_t(s,a)+{\alpha }_t\left(\overline{R}(s,a)+\gamma \underset{a^{\prime }\in \mathcal{A}}{max}{Q}_t(s_{t+1}^{\left(sa\right)},a^{\prime })\right),\hfill & (s,a)\in c(s_t,a_t),\hfill \\ Q_{t+1}(s,a)=Q_t(s,a),\hfill & (s,a)\notin c(s_t,a_t),\hfill \end{array}\right.\end{array}$$

(2)

where ${\left({\alpha }_t\right)}_{t\ge 0}$ is a sequence of suitably chosen learning rates, as in (1). The pseudo-code of QL-ES is provided in Algorithm 2.

Algorithm 2 QL-ES

Input: dataset $\mathcal{D}$, maximum iterations T, learning rates ${\left({\alpha }_t\right)}_{t\ge 0}$, equivalence structure $\mathcal{C}$

Initialization: $Q_0=0\in {\mathbb{R}}^{S\times A}$.

for  $t=0,1,2,\dots ,T$  do

Sample action $a_t\sim {\pi }_{\mathrm{b}}\left(s_t\right)$ and observe $s_{t+1}\sim P(·|s_t,a_t)$.

Find $c(s_t,a_t)$.

for $(s,a)\in c(s_t,a_t)$ do

$s_{t+1}^{\left(sa\right)}={\sigma }_{s,a}^{-1}\left({\sigma }_{s_t,a_t}\left(s_{t+1}\right)\right)$

Compute $Q_{t+1}$ using (2)

end for

end for

When the underlying MDP admits some equivalence structure, QL-ES performs multiple updates of Q-function at any slot, in contrast to structure-oblivious Q-learning that updates only the Q-function of the current state–action pair. Thus, we expect learning the optimal Q-function under QL-ES to be faster than Q-learning; this will be corroborated by the numerical experiments in Section 6. It is also worth mentioning that QL-ES is never worse than Q-learning, as for the trivial partition $\mathcal{C}=\mathcal{S}\times \mathcal{A}$, which holds for any finite MDP, QL-ES reduces to Q-learning.

Remark 1.

The multiple updates used in QL-ES can be straightforwardly combined with many other variants of Q-learning, such as Speedy Q-learning [46] and UCB-QL [41].

We finally remark that some works in the literature on Q-learning use learning rates of the form ${\alpha }_t=f\left(N_t(s_t,a_t)\right)$, where $N_t(s,a)={\sum }_{\tau =0}^{t-1}\mathbb{I}\{(s_{\tau },a_{\tau })=(s,a)\}$ and where f is some suitable function f satisfying the Robbins–Monro conditions, e.g., ${\alpha }_t=\frac{1}{N_t(s_t,a_t)+1}$. Such learning rates in the case of QL-ES can be modified to ${\alpha }_t=f\left(N_t\left(c(s_t,a_t)\right)\right)$, where for any $c\in \mathcal{C}$, $N_t\left(c\right):={\sum }_{\tau =0}^{t-1}\mathbb{I}\{(s_{\tau },a_{\tau })\in c\}$.

5. Theoretical Guarantee for QL-ES

In this section, we investigate the theoretical guarantee of QL-ES in terms of sample complexity in the PAC setting. Specifically, we are interested in characterizing the deviation between the optimal Q-function $Q^{\star }$ and its estimate $Q_T$ computed by QL-ES after T time steps. A relevant notion of deviation often studied in the literature (see, e.g., [37,38]) is the ${\ell }_{\infty }$-distance between $Q_T$ and $Q^{\star }$:

$$\parallel Q^{\star }-Q_T{\parallel }_{\infty }=\underset{s,a}{max}|Q^{\star }(s,a)-Q_T(s,a)|$$

(3)

which captures the worst error (with respect to $Q^{\star }$) among various pairs. One may study the rate at which the error function $\parallel Q^{\star }-Q_T{\parallel }_{\infty }$ decays as a function of T. Alternatively, one may characterize the PAC sample complexity defined as the number T of steps needed until $Q_T$ satisfies $\parallel Q^{\star }-Q_T{\parallel }_{\infty }\le \epsilon$ with probability at least $1-\delta$, for pre-specified $\epsilon$ and $\delta$. We consider the latter case.

Let us first recall the classic definition of cover time $t_{\mathrm{cover}}$, which is a standard notion in the literature on Markov chains as well as those studying theoretical guarantees of Q-learning (and its variants) [9,38,39]. Let $t_1\ge 0$ and let $t_2>t_1$ denote the first time step such that all state–action pairs are visited at least once with probability at least $\frac{1}{2}$. Then, the cover time $t_{\mathrm{cover}}$ is defined as the maximum value of $t_2-t_1$ over all initial pairs $(s_{t_1},a_{t_1})$. Note that $t_{\mathrm{cover}}$ depends on both the MDP M and the behavior policy ${\pi }_{\mathrm{b}}$. More precisely, it depends on the mixing properties of the Markov chain induced by ${\pi }_{\mathrm{b}}$ on M. Further, we have $t_{\mathrm{cover}}\ge SA$.

Next, we introduce a notion of cover time for equivalence classes, which is relevant to the performance analysis of QL-ES. We believe it can be of independent interest.

Definition 3.

Let M be a finite MDP and $\mathcal{C}$ be an equivalence structure in M. Given $t_1\ge 0$, let $t_2>t_1$ denote the first time step such that for each $c\in \mathcal{C}$, some state–action pair in c is visited at least once with probability at least $\frac{1}{2}$. Then, the cover time with respect to the equivalence structure $\mathcal{C}$ in M, denoted by $t_{\mathrm{cover},\mathcal{C}}$, is defined as the maximum value of $t_2-t_1$ over all initial choices of $c(s_{t_1},a_{t_1})$ (i.e., the class the initial pair $(s_{t_1},a_{t_1})$ belongs to).

The following theorem provides a non-asymptotic sample complexity for QL-ES. It concerns constant learning rates, i.e., ${\alpha }_t=\alpha$ for all $t\ge 0$, where $\alpha$ may depend on $\epsilon$ and $\delta$, among other things.

Theorem 1.

There exist some universal constants ${\kappa }_0,{\kappa }_1$ such that for any $\delta \in (0,1)$ and $\epsilon \in (0,\frac{1}{1-\gamma }]$, we have $\parallel Q^{\star }-Q_T{\parallel }_{\infty }\le \epsilon$ with probability greater than $1-\delta$, provided that the number T of steps and learning rate α jointly satisfy

$$\begin{array}{c}\hfill T\ge \frac{{\kappa }_0{t}_{\mathrm{cover},\mathcal{C}}}{{(1-\gamma )}^5{\epsilon }^2}{log}^2\left(\frac{CT}{\delta }\right)log\left(\frac{1}{{(1-\gamma )}^2\epsilon }\right),\alpha =min\left\{\frac{{\kappa }_1{(1-\gamma )}^4{\epsilon }^2}{{\gamma }^{2}log(CT/\delta )},\frac{1}{2}\right\}.\end{array}$$

A proof of this theorem is provided in Appendix A. Our proof is an adaptation of the of the proof of Theorem 2 in [9], which concerns the sample complexity of Q-learning.

Comparison with sample complexity of Q-learning. Theorem 1 tells us that the number of steps to have $\parallel Q^{\star }-Q_T{\parallel }_{\infty }\le \epsilon$ with high probability depends on $t_{\mathrm{cover},\mathcal{C}}{\epsilon }^{-2}{(1-\gamma )}^{-5}$ (up to some logarithmic factors), where $t_{\mathrm{cover},\mathcal{C}}$, defined in Definition 3, is the cover time with respect to $\mathcal{C}$. Comparing this result against the sample of complexity of Q-learning (e.g., Theorem 2 in [9]) reveals that using QL-ES yields an improvement over Q-learning by a factor of $\mathrm{const}.\times \xi$, where

$$\xi :=\xi (M,\mathcal{C},{\pi }_{\mathrm{b}}):=\frac{t_{\mathrm{cover}}}{t_{\mathrm{cover},\mathcal{C}}}.$$

This ratio $\xi$ is a problem-dependent constant (depending on both M and $(\mathcal{C},\sigma )$). It also depends on the behavior policy ${\pi }_{\mathrm{b}}$ in view of the definitions of the cover times. It is evident that $\xi \ge 1$ for any choice of M and $\mathcal{C}$. For a given MDP M, the ratio $\xi (M,\mathcal{C},{\pi }_{\mathrm{b}})$ can be numerically computed; we report numerical values of $\xi$ for several domains in Section 6.3. On the other hand, deriving the analytical bounds on the ratio $\xi (M,\mathcal{C},{\pi }_{\mathrm{b}})$ for any M appears to be complicated and tedious, if possible at all. Nonetheless, it is possible to construct simple problem instances, where one can derive analytical bounds on $\xi$.

Figure 2 portrays one such example; this example is a simple Markov chain but can be easily extended to become an MDP. Easy calculations show that $t_{\mathrm{cover}}=(S-1)/\delta$ and $t_{\mathrm{cover},\mathcal{C}}=1$ so that $\xi =(S-1)/\delta$. Hence, one here has $\xi =O\left(S\right)$. This simple example demonstrates that the gain of QL-ES over Q-learning in some domains could be as large as $O\left(S\right)$, the size of the state space. Additionally, Theorem 1 reveals that in such domains, the theoretical sample complexity bound of QL-ES does not depend on S but on C, the number of classes in $\mathcal{C}$.

We refer the reader to the results in Section 6.3, where we present numerical bounds on $\xi$ in some MDPs, which serve as the standard domain in the RL literature.

6. Simulation Results

This section is devoted to reporting numerical experiments conducted to examine the performance of QL-ES against the (structure-oblivious) Q-learning algorithm. First, we present the considered evaluation metrics and environments. Then we present numerical assessment of $\xi$ for these environments. Finally, we report empirical sample complexities of QL-ES and Q-learning in the environments.

6.1. Evaluation Metrics

We consider two evaluation metrics in the experiments:

(i)

Max-norm Q-value Error defined as $\parallel Q^{\star }-Q_t{\parallel }_{\infty }$;

(ii)

Total Policy Error defined as $\parallel {\pi }^{\star }-{\pi }_t^{\mathsf{greedy}}{\parallel }_1$, where ${\pi }_t^{\mathsf{greedy}}$ denotes the greedy policy w.r.t. $Q_t$, i.e., ${\pi }_t^{\mathsf{greedy}}\left(s\right):=arg{max}_a{Q}_t(s,a)$ for all s.

The metric (i), which is in line with the definition of sample complexity studied in Section 5, captures the maximum difference between $Q_t$ and $Q^{\star }$ over all state–action pairs and allows us to empirically study the convergence speed of $Q_t$ toward $Q^{\star }$. The second metric captures the quality of the estimate $Q_t$ in terms of inferred policies. Evidently, the quantity $\parallel {\pi }^{\star }-{\pi }_t^{\mathsf{greedy}}{\parallel }_1$ returns the number of states at which ${\pi }_t^{\mathsf{greedy}}$ prescribes a sub-optimal action. Hence, the metric (ii) may capture how bad the policy derived from $Q_t$ (i.e., ${\pi }_t^{\mathsf{greedy}}$) would be, compared to ${\pi }^{\star }$, had we stopped at time step t. Equivalently, we may compute the metric (ii) via

$$\sum _{s\in \mathcal{S}}\mathbb{I}\left\{Q_t(s,{\pi }^{\star }\left(s\right))-\underset{a\ne {\pi }^{\star }\left(s\right)}{max}{Q}_t(s,a)<0\right\}.$$

(4)

Working with (4) is preferred, as then, one may not worry about how ties (in $argmax$) are broken when either ${\pi }_t$ or ${\pi }^{\star }$ is not unique.

6.2. Environments

We consider two environments: RiverSwim and GridWorld. These are classical MDPs widely used in the RL literature. Both render suitability to demonstrate the numerical performance of QL-ES since each allows us to define a family of MDPs with progressive difficulty levels.

RiverSwim and variants. A generic RiverSwim MDP with L states is shown in Figure 1, which extends the classical 6-state RiverSwim presented in [45]. This MDP is constructed so that efficient exploration is required to obtain the optimal policy. The larger the number L of states, the more exploration is required. The L-state RiverSwim (with $L\ge 3$) admits an equivalence structure with $C=3$ regardless of L. We consider RiverSwim instances with various L so as to have MDPs with progressive difficulty levels while having a fixed number of classes. In some experiments, we consider a slightly modified version of RiverSwim, which we shall call Perturbed RiverSwim. It is identical to RiverSwim (Figure 1) except that in any state $s_i$, where $i<L$ is even, $p\left(s_i\right|s_i,\mathrm{R})=0.65$ and $p\left(s_{i+1}\right|s_i,\mathrm{R})=0.3$. It is clear that there are $C=4$ classes in a L-state Perturbed RiverSwim.

GridWorld. We also consider 2-room and 4-room grid-world MDPs with different grid sizes. Figure 3 shows a $7\times 7$ 2-room and a $9\times 9$ 4-room grid-worlds, respectively. In both environments, the agent starts at the upper-left corner (in red) and is supposed to reach the lower-right corner (in yellow), where it is given a reward of 1 and then sent back to the initial red state. At each step, the agent has four possible actions (hence, $A=4$): Going up, left, down, or right. Black squares indicate the wall where the agent is not able to penetrate through. After executing a given action, the agent has a probability of $0.1$ to stay in the same state, has a probability of $0.7$ to move to the desired direction, and has a probability of $0.06$ and $0.14$ to move to the other two possible directions. If the wall blocks the agent, it stays where it is, and the transition probability of the next state is added to that of the current state.

It is clear that the grid-world MDPs above admit some equivalence structure. In the case of 2-room (respectively, 4-room), the state–action space is of size 84 (respectively, 160), while the number of classes remains 8 in both. In

Table 1. State–action space and equivalence classes comparison in grid-world MDPs.

Environment	States	$7\times 7$	$9\times 9$	$11\times 11$	$20\times 20$	$50\times 50$	$100\times 100$
2-room	$SA$	84	172	292	1228	9028	$3.8\times {10}^4$
2-room	C	8	8	8	8	8	8
4-room	$SA$	80	160	272	1172	8852	$3.7\times {10}^4$
4-room	C	8	8	8	8	8	8

, we also present six examples of grid-world environments with walls defined according to the way mentioned above. In the introduced 2-room and 4-room MDPs, the number of state–action pairs changes with the increase in the grid size, while the number of classes remains fixed.

6.3. Bounds on the Ratio $\xi$

We recall from Section 5 that the theoretical gain of QL-ES over Q-learning in terms of sample efficiency is captured by the problem-dependent quantity $\xi =\frac{t_{\mathrm{cover}}}{t_{\mathrm{cover},\mathcal{C}}}$. In this subsection, we compute $\xi$ for the introduced environments with the aim of providing insights into the growth of $\xi$ as the number S of states grows. Specifically, we consider RiverSwim, Perturbed RiverSwim (introduced in Section 6.2), and GridWorld MDPs, each with growing number of states. In each case, we report empirical values for $t_{\mathrm{cover}}$ and $t_{\mathrm{cover},\mathcal{C}}$ together with the corresponding $95\%$ confidence intervals. The empirical $t_{\mathrm{cover}}$ is computed as the median value (across 100 independent runs for every possible initial state-action pair) of the number of steps it takes to discover all state-action pairs starting from a given initial state-action pair. A similar procedure is used for $t_{\mathrm{cover},\mathcal{C}}$.

Table 2. Empirical values of $t_{\mathrm{cover}}$, $t_{\mathrm{cover},\mathcal{C}}$, and ${\xi }_{\mathrm{LCB}}$ for RiverSwim with S states.

S	6	10	14	20
$t_{\mathrm{cover}}$	131, $\mathrm{CI}=[97,158]$	513, $\mathrm{CI}=[340,658]$	2529, $\mathrm{CI}=[1867,3196]$	12,792, $\mathrm{CI}=[7577,\mathrm{15,688}]$
$t_{\mathrm{cover},\mathcal{C}}$	12, $\mathrm{CI}=[9,16]$	33, $\mathrm{CI}=[26,38]$	56, $\mathrm{CI}=[46,66]$	113, $\mathrm{CI}=[94,133]$
${\xi }_{\mathrm{LCB}}$	$\frac{97}{16}\approx 6.1$	$\frac{340}{38}\approx 8.9$	$\frac{1867}{66}\approx 28.3$	$\frac{7577}{133}\approx 57.0$

,

Table 3. Empirical values of $t_{\mathrm{cover}}$, $t_{\mathrm{cover},\mathcal{C}}$, and ${\xi }_{\mathrm{LCB}}$ for Perturbed RiverSwim with S states.

S	6	10	14	20
$t_{\mathrm{cover}}$	116, $\mathrm{CI}=[97,136]$	557, $\mathrm{CI}=[351,655]$	2011, $\mathrm{CI}=[1616,2451]$	11,856, $\mathrm{CI}=[7282,\mathrm{17,103}]$
$t_{\mathrm{cover},\mathcal{C}}$	14, $\mathrm{CI}=[12,17]$	32, $\mathrm{CI}=[23,37]$	58, $\mathrm{CI}=[46,67]$	114, $\mathrm{CI}=[99,130]$
${\xi }_{\mathrm{LCB}}$	$\frac{97}{17}\approx 5.7$	$\frac{351}{37}\approx 9.5$	$\frac{1616}{67}\approx 24.1$	$\frac{7282}{130}\approx 56.0$

and

Table 4. Empirical values of $t_{\mathrm{cover}}$, $t_{\mathrm{cover},\mathcal{C}}$, and ${\xi }_{\mathrm{LCB}}$ for 2-room GridWorld with S states.

S	21	43	71	111
$t_{\mathrm{cover}}$	2164, $\mathrm{CI}=[1939,2355]$	5480, $\mathrm{CI}=[4890,5889]$	11,310, $\mathrm{CI}=[9817,\mathrm{12,735}]$	22,793, $\mathrm{CI}=[\mathrm{20,571},\mathrm{24,882}]$
$t_{\mathrm{cover},\mathcal{C}}$	205, $\mathrm{CI}=[125,251]$	418, $\mathrm{CI}=[329,481]$	877, $\mathrm{CI}=[613,1011]$	1338, $\mathrm{CI}=[1101,1551]$
${\xi }_{\mathrm{LCB}}$	$\frac{1939}{251}\approx 7.7$	$\frac{4890}{481}\approx 10.2$	$\frac{9817}{1011}\approx 9.7$	$\frac{20571}{1551}\approx 13.3$

summarize empirical values of $t_{\mathrm{cover}}$ and $t_{\mathrm{cover},\mathcal{C}}$ (together with the associated $95\%$ confidence intervals denoted by $\mathrm{CI}$) for RiverSwim, Perturbed RiverSwim, and 2-room GridWorld, respectively, with varying number of states in each case. In the case of GridWorld, we ran a uniform agent (sampling each action uniformly), wheres in RiverSwim MDPs, the agent samples R (resp. L) with probability $0.8$ (resp. $0.2$).

These results reveal that $t_{\mathrm{cover},\mathcal{C}}$ is much smaller than $t_{\mathrm{cover}}$ in all cases. Furthermore, they indicate that while $t_{\mathrm{cover}}$ grows rapidly as S increases (in any family of the MDPs considered), $t_{\mathrm{cover},\mathcal{C}}$ experiences a much smaller growth. As for the ratio $\xi$, we report ${\xi }_{\mathrm{LCB}}$ as the lower confidence bound obtained by dividing the lower value in the CI for $t_{\mathrm{cover}}$ by the upper value in the CI for $t_{\mathrm{cover},\mathcal{C}}$. This is a rather conservative estimate of the true $\xi$ but ensures that $\xi \le {\xi }_{\mathrm{LCB}}$ with probability at least $0.9$. The reported values demonstrate that in these environments, $\xi$ grows rapidly as the size of state space grows. This observation verifies that the theoretical gain of QL-ES over Q-learning can be significant.

6.4. Experimental Results with Exact Equivalence Structure

We now turn to reporting experimental results for QL-ES and Q-learning in RiverSwim and GridWorld. In the following figures, QL indicates the standard Q-learning algorithm (Algorithm 1). We used a constant learning rate $\alpha =0.05$ and $ϵ$-greedy policy (with values of $ϵ$ to be specified later). Furthermore, all the results are averaged over 100 independent runs, and the corresponding $95\%$ confidence intervals are shown.

Figure 4 presents the max-norm Q-value error and total policy error under both QL-ES and Q-learning in a 6-state RiverSwim, where we set $\gamma =0.9$ and $ϵ=0.5$. It is evident that QL-ES significantly outperforms Q-learning. The Q-value error under Q-learning decays at a very slow rate until about about $4\times {10}^5$ steps. After this step, the decay rate increases tangibly. In contrast, the Q-value error under QL-ES decays at a much faster speed. Under Q-learning, the total policy error remains above 5 until time step $4\times {10}^5$, which implies that only one state has learned its optimal policy. On the contrary, the total policy error under QL-ES drops to the vicinity of 0 very quickly. These results verify that the empirical gain of leveraging the equivalence structures in MDPs, in terms of the number of samples, can be significant.

To demonstrate the scalability of QL-ES, in Figure 5, we present the Q-value error under QL-ES in RiverSwim instances with 6, 20, and 40 states. As the figure shows, although the error in the 6-state RiverSwim starts decaying much earlier than the others, all of them exhibit a similar rate of decay. Moreover, the curves corresponding to 20-state and 40-state instances are almost indistinguishable. This result showcases MDPs where the sample complexity of QL-ES does not scale with the size of the state–action space and is mostly determined by the number of classes.

We now turn to the results for GridWorld MDPs. Figure 6 and Figure 7 show the results for QL-ES and Q-learning in 2-room and 4-room GridWorld MDPs, where we used $\gamma =0.85$ and $ϵ=0.2$ in the 2-room and $\gamma =0.85$ and $ϵ=0.3$ in the 4-room.

As in RiverSwim MDPs, QL-ES significantly outperforms Q-learning in the grid-world environments. For Q-learning, the Q-value error remains considerable, even for ${10}^6$ samples. Although both QL-ES and Q-learning do not fully learn an optimal policy by the end of the run, the total policy error decays much faster under QL-ES. Overall, the results demonstrate that exploiting equivalence structure is beneficial in grid-world MDPs. Comparing Figure 6 and Figure 7, it is evident that QL-ES still obtains relatively better performance than Q-learning with the increase in state space. Moreover, similar trends are expected when conducting this experiment in larger grid-world MDPs.

6.5. The Gain in the Case of $\theta$-Similar Pairs

We now investigate the case where the MDP may not admit any equivalence structure but admits $\theta$-similarity across its state–action space; see Definition 1. To this effect, we introduce Modified RiverSwim and Modified GridWorld defined as follows. The Modified RiverSwim is identical to 6-state RiverSwim (Figure 1), except that its non-zero transition probabilities under $(s_2,\mathrm{R})$ and $(s_4,\mathrm{R})$ are changed from $[0.05,0.55,0.4]$ to $[0.15,0.3,0.55]$. The Modified GridWorld is identical to $7\times 7$ 2-room grid-world, except that the non-zero transition probabilities under ($s_1$,down), ($s_5$,down), and ($s_7$,down) are set to $[0.6,0.05,0.2,0.15]$ instead of $[0.7,0.06,0.14,0.1]$. Modified RiverSwim admits an equivalence structure, but we remark that it satisfies $\theta$-similarity with $\theta =0.1$. Similarly, Modified GridWorld satisfies $\theta$-similarity with $\theta =0.1$. Figure 8 shows the results in Modified RiverSwim (with $\gamma =0.95$) and Modified GridWorld ($\gamma =0.85$). It is evident that in both cases, QL-ES still achieves smaller Q-value error than Q-learning.

6.6. The Impact of Partially Using the Structure

Considering MDPs with huge state–action spaces, it is necessary to take into account the feasibility of using only a few equivalent pairs. This thus naturally leads to the question as to whether only using a few equivalent pairs would lead to a reasonable performance gain. Therefore, here we investigate the convergence speed when choosing a subset of equivalent state–action pairs at each time step, rather than considering all the state–action pairs in the same class.

As shown in Figure 9, we can still obtain reasonable performance only considering a few equivalent pairs. The numbers in brackets represent how many equivalent pairs are used. In RiverSwim with 6 states ($SA=12$), the performance of QLES(3) and QLES(4) is comparable to QL-ES. Interestingly, we already observe a significant improvement over Q-learning using QLES(1), i.e., when using only one additional observation in the Q-learning update.

Meanwhile, the total policy error is less than or close to one. This shows that the optimal policy is correctly learned in almost all the states.

In 20-state RiverSwim ($SA=40$), algorithms with few equivalent pairs can still achieve excellent performance, albeit not as good as QL-ES. Additionally, as Figure 10 shows, using more equivalent pairs leads to better sample efficiencies in MDPs with large state space. Concerning QLES(3)-QLES(12), the Q-value error gets smaller when more equivalent pairs are used.

From the perspective of policy, even QLES(3) and QLES(6) manage to find optimal actions significantly faster than Q-learning. In addition, QLES(10) is far superior to QLES(6) because of the quite smaller Q-value error and total policy error. The results show that algorithms with a suitable number of equivalent pairs are sufficient to learn an optimal policy in most states reasonably fast.

7. Conclusions

We studied off-policy learning in discounted Markov decision processes, where some equivalence structure exists in the state–action space. We presented a model-free algorithm called QL-ES, which is a natural extension of the classical asynchronous Q-learning but capable of exploiting the equivalence structure. We presented a high-probability sample complexity bound for QL-ES, and discussed how it improves that of Q-learning. As demonstrated, there exist problem instances on which the improvement over Q-learning could be a multiplicative factor of S, the size of the state space. Through extensive numerical experiments in standard domains, we demonstrated that QL-ES significantly improves over (structure-oblivious) Q-learning. These results revealed that exploiting state–action equivalence favors faster convergence of the Q-function and policy learning in large MDPs. A limitation of our approach is the need for the prior knowledge on the structure. To the best of our knowledge, existing methods for learning the equivalence structure are all model-based. Hence, an interesting question is whether it is possible to exploit the equivalence structure without prior knowledge on the structure using only model-free algorithms. Devising such model-free algorithms (or otherwise establishing an impossibility result) is an interesting yet challenging topic for future work. Another interesting direction for future work is to investigate ways to combine the knowledge of the equivalence structure with function approximation methods. Finally, another avenue for future work is to study model-free algorithms for the regret minimization setting in average-reward MDPs (e.g., [47,48]).