An Experimental Study on State Representation Extraction for Vision-Based Deep Reinforcement Learning

Abstract

Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

1. Introduction

Deep Reinforcement Learning is an emerging subfield of Reinforcement Learning (RL) that relies on deep neural networks as a function approximator, enabling RL algorithms in complex environments. Over the last few years, end-to-end DRL algorithms have become increasingly more stable and efficient. Notable application successes include learning to play a variety of video games from raw pixels [1], continuous control tasks such as controlling a simulated car from a dashboard camera [2], and subsequent algorithmic developments and applications to agents that successfully navigate mazes and solve complex tasks from first-person camera observations [3,4,5], and robots that successfully grasp objects in the real world [6].

However, despite the impressive ability, learning policy directly from raw images is found to be relatively unstable and data-inefficient (usually needs millions of samples) than operating on coordinate-state based features [7,8]. On the one hand, unlike in classic reinforcement learning where human-crafted representation is used, vision-based DRL has to learn features directly from raw observations, in addition to policy learning; on the other hand, most RL approaches assume a fully observable state space, i.e., fully observable Markov Decision Processes (MDPs). However, this assumption is unworkable in real-world robotics due to factors including sensor sensitivity limitations and sensor noise and the lack of knowledge about whether the observation design is complete or not. Particularly in the case of learning directly from raw images, partial observability problem arises, which is defined as a partially observable MDP (POMDP), in which the agent can only sees a portion of the world state. Furthermore, there are numerous extrinsic factors (e.g., hyperparameters or codebases) and intrinsic elements (e.g., effects of random seeds or environment properties) that might affect the performance of DRL [9]. As a result, vision-based DRL typically suffers from slow learning speeds and frequently requires an excessive amount of training time and data to attain desired performance, making it unsuitable to real-world situations where data collection is difficlut and expensive.

For this reason, most of the present DRL algorithms are still limited to solving tasks with state spaces of relatively low dimensionality (less than 10 intrinsic dimensions for value-function based methods and less than 100 for policy gradient methods [10]). In addition, when the ground-truth state information cannot be easily accessed, it is inevitable to extract the embedded states from the original observations to learn desirable policies from DRL algorithms. To overcome this issue, state representation learning (SRL) [11] was proposed which aims at learning to extract the state information which can be used for the policy optimization.

As shown in Figure 1. This paper studies the influence factors (listed on the bottom left corner of this figure) of SRL, and their effects on different kinds of RL methods (listed on the bottom left corner of this figure). In the setting of SRL, instead of getting a low-dimensional coordinate state $s_t\in S$ at time t, the agent receives rather a high-dimensional observation $0_t\in O$, which is a rendering of potentially incomplete view of the corresponding state $s_t$ of the environment [12]. In principle, the agent should be able to learn representations that can recover the task-related state information. Suppose the state information is covered by pixels. In that case, it should be possible to learn from pixels as efficient as from ground-truth states given the reasonable representation.

There are many different SRL methods proposed for this objective, which have made remarkable progress, such as autoencoders, learning forward models, inverse dynamics and learning from robotic priors from the state characteristics [11]. Furthermore, with the tremendous development in label-efficient learning for image classification using contrastive unsupervised representations [13,14] and data augmentation [15,16], Laskin et al. [17,18] combined contrastive learning and data augmentation techniques from computer vision with model-free RL to demonstrate significant sample-efficiency gains on standard RL benchmarks. Besides, transfer learning has improved vision-based RL performance between different tasks further. PAD [19] performs unsupervised policy adaption with a test-time auxiliary task (e.g., inverse dynamic prediction) to fine-tune the visual encoder of the policy network. By means of unsupervised keypoint detection [20], a clean foreground mask could be reconstructed from noise-augmented observations and keep the RL policy learning between different tasks stable and fast [21].

So far, however, there has been little discussion about the influence factors (as shown in Figure 1) and evaluation metrics of state representation learning. Most previous studies in this field only focus on improving the effectiveness of SRL by means of designing new mechanisms and auxiliary tasks. The evaluation metric is also limited to RL’s performance, which means the RL agent’s learning process is the only criteria to evaluate the SRL approaches in most cases. The diversity in methods and applications makes this field lack statistically relevant results to analyze the influence factors of the SRL methods, which creates the potential for misleading reporting of results and the demands for tighter standardization of experimental reporting. More importantly, reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining the progress of SRL.

Therefore, this paper is set out to analyze the effects of SRL influence factors on vision-based DRL (Section 2). We summarized the criteria that could be used for evaluating SRL performance and compared the SRL performance on different kinds of RL algorithms (including value-based and policy-based methods):

• Firstly, in order to give more statistically representative experimental results and quantitative analysis, we adopted 8 different tasks, including mobile robot navigation and robot arm manipulation environments (Section 3). The complexity of the tasks and the observations increases gradually;
• Secondly, three evaluation criteria, including the quantitative evaluation with reinforcement learning performance, are summarized to evaluate the SRL performance based on the extensive investigation of the related works in Section 4.
• Thirdly, a large number of contrastive experiments were conducted using the control variates approach to investigate SRL influence factors such as hyperparameter, embedding network design, embedded dimension, regularization methods, and sample quality (Figure 1). In each part of the experimental analysis, we posed questions about major factors affecting final performance. We discovered that state embedded dimension and sample quality are crucial in the SRL model, and carefully adjusting the other illustrated factors can assist improve the SRL model’s representative property to a some extent (Section 5).
• Finally, we investigated the effects of these variables in reported results through a representative set of RL algorithms. Both value-based and policy-based methods with neural network function approximators, such as DQN, A2C, ACKTR, PPO, TRPO are chosen for the RL experiments (Section 5.7).

The experimental results are discussed in Section 6. Based on our experiments, we conclude with possible recommendations and points of discussion for future works to ensure that SRL continues to matter (Section 7). To the best of our knowledge, the experimental work presented in this paper provides one of the first investigations into adjusting the influence factors and designing efficient SRL models.

2. Problem Formulation

In this section, we will introduce the preliminary knowledge of RL and SRL methods which are integrated in our framework. As shown in Figure 2, when we describe the network used in DRL, it could be separated into two parts for the simplicity of explanation: the first part is used for representation learning, while the second part takes the learned representation as input and learns the value function and the target policy. State representation is learnt to improve two kinds of objectives: performance on the RL task and the auxiliary task.

More details on the auxiliary tasks are given in Section 2.2.2 and Section 5.6. There are many kinds of auxiliary tasks that can be added to the SRL scheme, such as image reconstruction, contrastive learning and model prediction. More details on the auxiliary tasks are given in Section 2.2.2. The dash green color means some SRL methods such as CURL can backpropagate the gradient from the RL loss function to the SRL network. The notations adopted in this paper and their definitions are list in

Table 1. Summary of the notations in this paper.

Symbol	Definition
$\pi$	policy
$r_t$	reward at time t
$o_t$	observation at time t
$\mathcal{O}$	observation space
$s_t$	ground-truth state at time t
$\mathcal{S}$	ground-truth state space
$\varphi \left(s\right)$	feature vector representing state s
$\Phi$	state representation space
$V_{\pi }\left(s\right)$	value of state s under policy $\pi$
$Q_{\pi }(s,a)$	value of taking action a in state s under policy $\pi$
J	The objective function (expected total reward)
$G_t$	return (accumulated rewards) following time t
$\theta$	parameter vector of policy ${\pi }_{\theta }$

.

2.1. Reinforcement Learning

2.1.1. Preliminary Knowledge

In RL, an agent learns an optimized policy through interacting with the environment. This process can be formulated as Markov Decision Processes (MDPs). An MDP can be defined as a 4-tuple $(S,A,T,R)$, where S represents the state space, A denotes the action space, transition probability $T:S\times A\mapsto S$ denotes the state transition probability, and R represents the reward function. Specifically, at time step t, the agent faces a state $s_t\in S$, and it follows the policy $\pi (a_t\mid s_t):s_t\mapsto a_t$ to choose an action $a_t\in A$. Then the agent receives a reward $r_t$, which could evaluate the performed action. After that, the agent reaches the next state $s_{t+1}$. The aforementioned process will break when the terminal condition is satisfied.

The return $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}$ is the total accumulated rewards at state $s_t$ with discount factor $\gamma \in (0,1]$, which is a hyperparemeter to adjust the impact of actions in the future. The goal of RL algorithms is to maximize the expected return $R_t$ by updating the policy $\pi$. The value function of state $s_t$ under policy $\pi$ is usually defined as:

$$V^{\pi }\left(s_t\right)=\mathbb{E}\left[G_t|s_t=s\right].$$

(1)

Similarly, the value of selected action $a_t$ in state $s_t$ under policy $\pi$ is defined as:

$$Q^{\pi }(s_t,a_t)=\mathbb{E}\left[G_t|s_t=s,a\right],$$

(2)

which is called action-value function. The advantage function estimates the average advantage value of selected action $a_t$ in state $s_t$ under policy $\pi$ and it is defined as:

$$A^{\pi }\left(s_t,a_t\right)=Q^{\pi }\left(s_t,a_t\right)-V^{\pi }\left(s_t\right).$$

(3)

2.1.2. Value-Based and Policy-Based RL Methods

There are two important categories of RL algorithms: value-based RL methods and policy-based RL methods. Policy-based RL methods explicitly build a representation of the policy $\pi (a_t\mid s_t;\theta )$ and update the parameters $\theta$. In contrast to policy-based RL methods, value-based RL methods don’t build any explicit policy, but a value function $V(s;\theta )$ or an action-value function $Q(s,a;\theta )$ with parameters $\theta$. The policy is implicit and can be derived directly from the value function (pick the action with the best value). Actor-critic methods mix policy-based methods and value-based methods, which often achieve better results than both policy-based methods and value-based methods.

In actor-critic methods, the actor is a parameterized policy that determines how actions are selected. Policy gradient methods are usually employed to learn the parameters. We consider a stochastic policy gradient with respect to the policy parameters $\theta$. The following equation gives the updating rule for $\theta$:

$$\theta =\theta +\alpha \nabla J\left(\theta \right),$$

(4)

where $J\left(\theta \right)$ is the expected total reward and $\nabla J\left(\theta \right)$ can be calculated by:

$$\nabla J\left(\theta \right)={\mathbb{E}}_t\left[{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right)A\left(s_t,a_t\right)\right]$$

(5)

The advantage function captures how better an action is than the others at a given state to evaluate the current policy. It is found that using the advantage function has a better performance than other metrics like the Q values [22].

With sufficient samples, the expectation of $A\left(s_t,a_t\right)$ can be evaluated accurately. Unfortunately, in many real-world tasks, it is impractical to collect such a great amount of samples, leading to the high variance of $A\left(s_t,a_t\right)$. Therefore, it is necessary to use the value-based critic to estimate the $A\left(s_t,a_t\right)$. If two critic networks are employed to estimate the value function $Q^{\pi }\left(s_t,a_t\right)$ and $V^{\pi }\left(s_t\right)$ respectively, it will lead to more bias. Thus, Time-difference method is often used. In fact, $Q^{\pi }\left(s_t,a_t\right)$ and $V^{\pi }\left(s_t\right)$ satisfy the following equation:

$$Q^{\pi }\left(s_t,a_t\right)=\mathbb{E}\left[r_t+V^{\pi }\left(s_{t+1}\right)\right]$$

(6)

Though $r_t$ is a random variable, the variance of $r_t$ is much smaller than that of $A^{\pi }\left(s_t,a_t\right)$ when $A^{\pi }\left(s_t,a_t\right)$ is estimated directly. So the Equation (6) can be rewritten as:

$$Q^{\pi }\left(s_t,a_t\right)\approx r_t+V^{\pi }\left(s_{t+1}\right)$$

(7)

Therefore, it is possible to use only one critic network to estimate $A\left(s_t,a_t\right)$:

$$A^{\pi }\left(s_t,a_t\right)=r_t+V^{\pi }\left(s_{t+1}\right)-V^{\pi }\left(s_t\right)$$

(8)

2.2. State Representation Learning

2.2.1. Preliminary Knowledge

State representation learning (SRL) aims at learning compact representations from raw observations without explicit supervision. Many different SRL approaches have been proposed [11], which plays an important role in deep reinforcement learning. SRL is utilized to learn a transformation $\Pi$ from the observation space $\mathcal{O}$ to the latent state space $\Phi$. Then, a policy $\pi$, which takes state representation $\varphi \in \Phi$ as input and outputs action $a\in \mathcal{A}$, is learned to solve the task:

$$\mathcal{O}\underset{SRL}{\overset{\Pi }{\to }}\Phi \underset{RL}{\overset{\pi }{\to }}\mathcal{A}$$

(9)

The state representation contains information extracted from states—the description of the current situation given by the environment. Therefore, a high-quality representation should only keep the valuable information for the task and reduce the search space, thus contributing to address two main challenges of RL: instability and sample inefficiency. It is not only essential to build a robust reinforcement learning agent but also can help improving learning efficiency. Moreover, a state representation learned for a particular task may be transferred to related tasks, speeding up learning in multiple tasks’ settings.

Many of the previous works in representation learning for reinforcement learning focused on designing fixed-basis architectures to achieve desirable properties, such as orthogonality. Most of these properties are common to the general machine learning setting. Many approaches either use or search for orthogonal or decorrelated features, such as orthogonal matching pursuit [23], Bellman-error basis functions [24], Fourier basis [25] and tile coding [26,27]. Prototypical input matching methods have been extensively explored, as in kernel methods [28], radial basis functions [26].

In contrast, the idea behind DRL is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation—excellent representations will arise via appropriate training schemes, where a good representation is defined by success on some task. Recent developments in representation learning explore a different perspective: we should let the training data dictate the properties of the representation through gradient descent. This view is widely held and is reflected in specifying training regimes, including multi-task training [29], auxiliary loss constructing (i.e., autoencoding, next observation prediction and pixel control) [3,30], and training on distribution of problems like meta-learning [31,32,33].

2.2.2. Auxiliary Tasks

Auxiliary tasks can be added to alter the complexity and the structure of the problem settings. By means of dividing the last layer of the neural network into several heads, it is possible to give various tasks to separate heads and then solve them collaboratively. The purpose of including auxiliary tasks into SRL is to help the learner develop a more accurate representation of the proeblem at hand. Several studies have shown that certain auxiliary tasks linked to the primary task are beneficial, such as learning an additional strategy to alter pixels in the observation maximally or maximally activate each bit in the representation [3].

Auxiliary tasks such as predicting the future conditioned on the past observations and actions [3,34,35,36] are a few representative examples of using auxiliary tasks to improve the sample-efficiency of model-free RL algorithms. The future prediction is either done in a pixel space [3] or latent space [35]. The sample-efficiency gains from reconstruction-based auxiliary losses have been benchmarked in [3,29,37]. Recently, contrastive learning has been used to extract reward signals in the latent space [38,39,40]; and study representation learning on Atari games [41].

In the next parts, we present approaches of SRL that are implemented in this paper. Each technique is not mutually exclusive and may be used in conjunction with others to generate new models.

2.2.3. Forward Model and Inverse Model

One critical element to encode for RL is the controlled agent’s state. It is equivalent to the robot position in the context of goal-based robotics activities. A straightforward approach is to use the forward model and inverse model.

A forward model that predicts state $s_{t+1}$ given state $s_t$ and action $a_t$, can be interpreted as learning dynamics of the world. Constraints may be introduced to the state representation by limiting the forward model, for example, by requiring the system to follow linear dynamics [42].

Another approach is to learn an inverse model [43], which predicts the taken action $a_t$ given two successive states $s_t$ and $s_{t+1}$. This requires embedded states to encode information about the dynamics in order to retrieve the action $a_t$, which can make such a transition. However, the features extracted by an inverse model are not always sufficient: in our goal-based experiments, they cannot embed the target’s position because the agent is unable to act on it. For this reason, additional objective functions are required to encode the target object’s position. Two of them are discussed in the new two sections: minimizing image reconstruction errors (autoencoder model) and minimizing reward prediction errors (reward prediction model).

2.2.4. Autoencoder

One intuitive step to get a state representation is to condense the observation into a low-dimensional embedded state that can be reconstructed. Because this method excludes potential actions, it does not make use of the robotic environment. We compared three kinds of autoencoders in this paper, including original Autoencoders [44], Variational Autoencoders (VAE) [45]—autoencoders that enforce the latent variables to follow a given distribution, and Denoising Autoencoder (DAE) [46] which could enhance the flexibility of data stream method in exploiting unlabeled samples.

Autoencoders tend to encode only aspects of the environment that are salient in the input image. This means they are not task-specific: relevant elements can be ignored, and distractors (unnecessary information) can be encoded into the state representation. They usually demand more dimensions in order to encode a scene (e.g., in our experiments, it requires more than 10 dimensions to encode a 2D position of the mobile robot correctly).

2.2.5. Reward Prediction

The objective of a reward prediction model [34] leads to state representations that are specially appointed by the goal of the task. This, however, does not imply that the state space must be disentangled or have any certain structure.

2.2.6. Robotic Prior Knowledge

Robotic prior knowledge about the dynamics or physics of the world may be used to construct an appropriate representation of states. This kind of knowledge may account for temporal continuity or causality principles that reflect an agent’s interactions with its surroundings [47,48].

2.3. Combination SRL Model and Splits SRL Model

There are generally two types of SRL model for learning state representations: the combination srl model and the splits srl model, which will be introduced briefly:

2.3.1. Combination SRL Model

In SRL Combination model, the combination of different objective functions is done by averaging the corresponding extracted features on a single embedding.

Combining objectives makes it possible to share the strengths of each model. In our application example, the previous sections suggest that we should mix objectives to encode both robot and target positions. The simplest way to combine objectives is to minimize a weighted sum of the different loss functions, i.e., reconstruction loss ${\mathcal{L}}_{\mathrm{reconstruction}}$, inverse dynamics prediction loss ${\mathcal{L}}_{\mathrm{inverse}}$ and reward prediction losses ${\mathcal{L}}_{\mathrm{reward}}$:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{combination}}=& w_{\mathrm{reconstruction}}·{\mathcal{L}}_{\mathrm{reconstruction}}+w_{\mathrm{inverse}}·{\mathcal{L}}_{\mathrm{inverse}}\hfill \\ \hfill & +w_{\mathrm{reward}}·{\mathcal{L}}_{\mathrm{reward}}\hfill \end{array}$$

(10)

Each weight reflects the proportional significance we place on certain objectives. We selected the weights such that the gradients are comparably important.

2.3.2. Splits SRL Model

Combining objectives into a single embedding is not the only option to have features that are sufficient to solve the tasks. Stacking representations, which also favors disentanglement, is another way of solving the problem. In SRL Splits model, the model described that combines different objective functions using splits of the state representation. The intrinsic idea of the SRL Splits model is that the state representations are divided into several parts, where each part optimizes a fraction of the objectives.

This stacked representation learning model prevents objectives that can be opposed from cancelling out and allows a more stable optimization. This process is similar to training several models but with a shared feature extractor, that projects the observations into the state representation.

3. Environment Description

In this section, we will describe the task environments with incremental difficulty. The mobile robot navigation environment and the Kuka robot arm environment with OpenAI Gym [49] interface are adopted, making integration with DRL algorithms easy.

To assure fairness, we execute five experiment trials for each evaluation, each with a distinct random seed (all experiments use the same set of random seeds). In all cases, we highlight important results, with complete descriptions of experimental setups, additional learning curves and statistical tables included in the supplemental materials. Unless otherwise mentioned, we use default settings whenever possible.

As shown in Figure 3, the robotic tasks proposed in this paper are variations of two environments: A 2D environment with a mobile robot and a 3D environment with a robotic arm. Each environment may have a continuous or discrete action space, with sparse or shaped rewards, enabling us to cover a wide range of scenarios. The ground-truth state is defined in each scenario: the absolute robot position in static scenarios or the relative position in moving goal scenarios. The tasks have incremental difficulty: the minimal number of variables for describing each task (minimal state dimension for solving the task with RL) is increasing from 2 (mobile robot 1D with random target) to 5 (robotic arm with random target). The detail description for each task will be introduced in Section 3.

3.1. Mobile Robot Navigation Environment

This setting simulates a navigation task using a small car resembling the task of [47], with either a cylinder or a horizontal band on the ground as a goal, which can be fixed or moving from episode to episode.

As it shows in Figure 3, in the mobile robot 1D navigation task, the robot moves along a straight line, and the target is a yellow cylinder; In the other three tasks of this environment, the robot moves on a 2D plane. The target in the mobile robot line-target task is a yellow band. The targets in the mobile robot 1-target task and 2-target task are yellow cylinder and yellow/red cylinders, respectively.

The action space of the mobile robot in 1D navigation task and 2D navigation tasks has 2 dimensions (left, right) and 4 dimensions (forward, backward, left, right), respectively (

Table 2. The property descriptions of two simulation environments.

Dataset	Reward	Action Space
Mobile Robot	Sparse	4 (2 *), Discrete (left, right, forward *, backward *)
Robotic Arm	Sparse	5, Discrete, (forward, backward, left, right, down)

* is for mobile robot 1D target task.

). The ground truth states in the 1D task and 2D task are the absolute coordinates of the robot and the target in static scenarios or the relative position in moving target scenarios. The reward function is defined as follows:

$$r=\left\{\begin{array}{cc}+1& whenthecarreachesthetarget\\ -1& whenthecarhitsthewall\\ 0& otherwise\end{array}\right.$$

(11)

3.2. Kuka Robot Arm Environment

This setting simulates a Kuka robotic arm fixed on a table, with the task of pushing a button that may move or not in between episodes. The arm can be controlled either in the x, y and z position using inverse kinematics, or directly controlling the joints.The action space of the Kuka robot arm has 5 dimensions (Table 2). The ground truth state is the absolute coordinates of the robot and the target(s) in static scenarios or the relative position in moving target scenarios. The episode will terminate when the arm hits the table. The reward function is defined as follows:

$$r=\left\{\begin{array}{cc}+1& whenthearmpushesthebutton\\ -1& whenthearmhitsthetable\\ 0& otherwise\end{array}\right.$$

(12)

3.3. State Representation Learning Pipeline

The complete pipeline consisted of (1) training the state representation networks with different settings; (2) RL performance evaluation on tasks with fixed representations; (3) SRL influence factors’ property evaluation. We save the learned representation by saving parameters $\varphi$, and then evaluate these fixed representations in terms of properties for learning performance in SRL and RL.

In each scenario, RGB images from a fixed position are defined as observations for state representation learning. Note that apart from providing all described environments, mobile robot 2D navigation datasets use 4 discrete actions (right, left, forward, backward); robot arms use one more action (down).

4. Evaluation Criteria

In this section, we summarized three evaluation criteria to analyze the representation learning influence factors and compare their effects on the final performance of reinforcement learning. The first two kinds of criteria come from the field of pattern recognition. They do not rely on specific RL algorithms or the policy training process, but only focus on the intrinsic structures of the learned representations and their relationship with the ground-truth states. The final criterion is the RL performance applied to the task, which should be the most convincing evaluation criterion of the extracted state representations. However, this approach is relatively computational expensive because it can only be obtained after training the SRL network and RL network.

4.1. K-Nearest Neighbors Evaluation Criterion (KNN)

The K-Nearest Neighbors ($KNN$) Evaluation Criterion is a criterion for evaluating embedded representations’ quality based on the Nearest-Neighbours method [50], it can help us get the nearest neighbors of an image in the latent representation space because a good representation should exhibit local coherence, which implies that the ground truth states associated with it should be near. Lesort et al. [51] derives a more typical metric named $\mathit{KNN}-\mathit{MSE}$ on the basis of $KNN$. A low $\mathit{KNN}-\mathit{MSE}$ value indicates that a neighbor in the ground truth state space is still a neighbor in the learned representation space, preserving local coherence.

For a given state observation o, we begin by finding its associated state’s $\varphi \left(s\right)$ closet neighbors $\varphi {\left(s\right)}^{\prime }$ in the learned state space $\Phi$ by means of $KNN$. Then we project them into the ground-truth state space $\mathcal{S}$. After that, in the latter space $\mathcal{S}$, we compute the average distance to its neighbors:

$$\mathit{KNN}-\mathit{MSE}\left(\varphi \left(s\right)\right)=\frac{1}{k}\sum _{\varphi {\left(s\right)}^{\prime }\in KNN(\varphi \left(s\right),k)}{∥s-s^{\prime }∥}^2,$$

(13)

where $\mathit{KNN}-\mathit{MSE}\left(\varphi \right(s),k)$ returns the k nearest neighbors $\varphi {\left(s\right)}^{\prime }$ of $\varphi \left(s\right)$ in the learned state space $\Phi$, s is the ground-truth state associated to $\varphi \left(s\right)$, and $s^{\prime }$ is the one associated to $\varphi {\left(s\right)}^{\prime }$.

4.2. Ground Truth Correlation (GTC)

A Pearson correlation coefficient’s matrix is computed for each dimension pair $(s,\varphi (s\left)\right)$:

$${\rho }_{\varphi \left(s\right),s}=\frac{\mathbb{E}\left[\left(\varphi \left(s\right)-{\mu }_{\varphi \left(s\right)}\right)\left(s-{\mu }_s\right)\right]}{{\sigma }_{\varphi \left(s\right)}{\sigma }_s}$$

(14)

where s is the ground truth state, $\varphi \left(s\right)$ is the learned state. ${\mu }_s$ and ${\sigma }_s$ are the mean and standard deviation of state s respectively.

We can visualize the correlation matrix to quantitatively assess the ability of a model to encode relevant information in the states learned. However, the visualization idea is impractical when meeting high-dimensional spaces such as those in our experiments. As a result, Raffin et al. proposed another two metrics: Ground Truth Correlation ($GTC$) and the mean of that—$GT{C}_{mean}$ [52]. They can be used to assess a model’s capacity to encode important information, measuring the similarity between the state representation $\varphi \left(s\right)$ and the ground-truth state s. The only difference between $GTC$ and $GT{C}_{mean}$ is that that $GT{C}_{mean}$ just need one scalar value to evaluate the learned state. More specifically, for each component $s_i$ of s, $GT{C}_i$ gives the maximum absolute correlation value between $s_i$ and any component of the predicted states $\varphi \left(s\right)$:

$$GT{C}_i=\underset{j}{max}\left|{\rho }_{\varphi \left(s\right),s}(i,j)\right|\in [0,1]$$

(15)

$$GT{C}_{\mathrm{mean}}=\mathbb{E}\left[GTC\right]$$

(16)

with $i\in \left[0,\left|s\right|\right]$, $j\in \left[0,\left|\varphi \right(s\left)\right|\right]$, $s=\left[s_1;\dots ;s_n\right]$, and $s_k$ being the $k^{th}$ dimension of the ground truth state vector. Taking the Mobile Robot 2D navigation with one random target task as an example, the the ground-truth state have a dimension of 4 ($\left|s\right|=4$), which is made up of the two-dimensional robot position and the two-dimensional target position.

4.3. Quantitative Evaluation with Reinforcement Learning

The reinforcement learning performance in the representation space is the most essential criterion for evaluating SRL approaches. In the experiments of this paper, five DRL algorithms (DQN, A2C, ACKTR, PPO, TRPO) including both value-based and policy-based methods are implemented and compared. The detail training configurations for each algorithm can be found in Appendix A.

5. Experimental Analysis

In this section, we pose several questions about the influence factors of SRL. Then we performed a set of experiments designed to provide insights into these questions. In particular, we investigate the influence of hyperparameters, embedding network architecture, embedded state dimension, sample quality, regularization approaches and SRL methods. In most of these experiments, PPO was chosen as the baseline RL algorithm because it works well across environments without much hyperparameter tuning.

To evaluate the properties of the embedded state representations, in the experiments except that in Section 5.4, we collect pixel data offline by doing a 50 episodes exploration, which generate more than 20,000-step transitions. The datasets that are composed of state observations (224 × 224@3 pixels) are collected from a random policy. In Section 5.4, the random policy was substituted by pre-trained PPO policies to analyze the influence of sample quality.

Each state representation is learned using 20,000 samples. This dataset is separated into the training set $(80\%)$ and validation set $(20\%)$. We kept for each method the model with the lowest validation loss during the 30 training epochs. We used the same network architecture from [52] for the most of models.

5.1. Hyperparameters

What is the magnitude of the influence that the hyperparameter settings can have on baseline performance?

Hyperparameters tuning plays an essential role in eliciting the best results from many algorithms. The choice of hyperparameters is crucial for reproducible end-to-end training [17]. However, the optimal hyperparameter configuration is often not consistent in related literature, and the range of values considered is often not reported. Sometimes, a high level of stochasticity appears due to random seeds [9]. It is dificult to assess the performance of high-level algorithmic ideas without understanding lower-level choices, as performance can be heavily influenced by hyperparameters tuning and implementation-level details, making it hard to attribute progress in representation learning and deep reinforcement learning. Thus, it slows down further research [53,54,55]. Furthermore, poor hyperparameter selection can be detrimental to a fair comparison against baseline algorithms.

In two mobile robot navigation challenges, we investigate three sets of hyperparameter choices for the kernel number in each convolutional layer and the corresponding pooling layer of the classic autoencoder neural network. The specific settings are shown in Figure 4, the total number of neural network hyperparameters in these three configurations are 1.8k+, 87.9k and 789.0k, respectively.

Results: The evaluation criteria such as $GTC$, $GT{C}_{mean}$, $\mathit{KNN}-\mathit{MSE}$ and average total reward can been seen in

Table 3. The influence of the network configuration in mobile robot navigation task.

Task	Network Configuration	Correlation					$\mathit{KNN}-\mathit{MSE}$	Average Total Reward PPO (1e6 Timesteps)
		Max Correlation Matrix				Mean
1D-Nav.	Config 1 (1.8 k*)	0.7821	2.9816e-15	None	None	0.3910	0	238.41 ± 3.72
	Config 2 (87.9 k)	0.8563	3.2195e-15	None	None	0.4282	0	238.35 ± 4.68
	Config 3 (789 k)	0.5612	2.3240e-15	None	None	0.2806	0	238.21 ± 3.76
2D-Nav.	Config 1 (1.8 k*)	0.4797	0.5046	0.9999	0.9999	0.7461	0.00063	229.04 ± 4.90
	Config 2 (87.9 k)	0.3303	0.4587	0.9999	0.9999	0.6973	0.00066	227.30 ± 5.38
	Config 3 (789 k)	0.2176	0.3834	0.9999	0.9999	0.6503	0.00064	227.05 ± 6.09

. The related learning performance of DQN and PPO methods is shown in Figure 5. The experiment results show that the effects of neural network hyperparameters are not consistent across algorithms or environments. However, the neural network hyperparameters do significantly affect learning performance, especially in the early stage of training. The performance of learnt latent feature representations is also related to the selected DRL algorithm. Usually, neural networks with more hyperparameters and deeper layers can show better function approximation and feature extraction ability. However, the vanishing gradients problem, exploding gradients problem and overfitting problem are easier to be encountered [56].

5.2. Network Architecture

How can the choice of network architecture for the SRL encoders affect results?

In this section, we utilized three different autoencoder methods to compare the influence of encoding network architecture, including original autoencoder, variational autoencoder (VAE) [45] and denoising autoencoder (DAE) [46]. Traditional autoencoder accepts and compresses input, then recreates the original input. This is an unsupervised technique because all you need is the original data, without any labels of known, correct results; A variational autoencoder assumes that the source data has some underlying probability distribution (such as Gaussian) and then attempts to find the parameters of the distribution; Denoising autoencoders are an example of deep architectures that are designed to recover noisy data, trying to achieve a good representation by changing the reconstruction criterion.

Results:Figure 6 shows how significantly learning performance can be affected by simple changes to the embedding network architecture. The autoencoder has mixed results: it solves all environments, yet it sometimes under-performs navigation tasks. When we explored the latent space using the S-RL Toolbox visualization tools [52], we noticed that one state space dimension can act on both robot and target positions in the reconstructed image. In addition, this approach does not use additional information that the environment provides, such as actions and rewards, leading to a latent space that may lack informative structure.

5.3. Embedded State Dimension

How to choose the embedded dimension of the SRL model?

To assess how the choice of embedded dimension can affect the RL algorithm performance, we use our aforementioned default set of hyperparameters except of the embedded dimension across an extended suite of discrete robotic tasks and study how well the RL algorithms perform across the different embedded states.

The following two tasks with OpenAI Gym interface are utilized for the experiments: mobile robot 1D navigation task and mobile robot 2D navigation with random-target task. The dimensions of the ground-truth state space (i.e., robot and target positions) in these two tasks are 2 and 4 separately. The dimensions of the action space are 2 and 4, respectively. The complexity of the image observations also increases, which makes the state representation learning process gradually difficult.

Results:

Table 4. The influence of the embedded dimension in Mobile Robot environment. The average total reward is calculated among the last 100 episodes of ${10}^6$ training timesteps.

Task	Embedded Dimension	Ground Truth Correlation (GTC)					$\mathit{KNN}-\mathit{MSE}$	Average Total Reward
		Max Correlation Matrix				Mean		DQN	PPO
1D-Nav.	1	0.6304	2.4059e-15	None	None	0.3152	4e-05	137.45 ± 75.10	238.95 ± 4.29
	2	0.7808	1.7720e-15	None	None	0.3904	0	201.39 ± 50.51	239.06 ± 4.31
	3	0.8609	3.6633e-15	None	None	0.4304	0	238.29 ± 4.83	237.54 ± 5.02
	4	0.5340	2.8306e-15	None	None	0.2670	0	237.18 ± 8.39	214.43 ± 42.96
	10	0.4186	2.5974e-15	None	None	0.2074	0	237.43 ± 5.66	230.76 ± 26.32
	20	0.5624	2.0297e-15	None	None	0.2812	0	235.12 ± 16.97	238.51 ± 4.60
	50	0.6885	2.2200e-15	None	None	0.3443	0	235.85 ± 12.09	238.69 ± 4.62
	100	0.8709	2.3983e-15	None	None	0.4355	0	225.80 ± 12.51	237.97 ± 5.23
	200	0.8563	3.2195e-15	None	None	0.4282	0	213.4 ± 40.65	238.35 ± 4.68
	1000	0.7564	4.0314e-15	None	None	0.3782	0	227.10 ± 25.33	222.80 ± 34.69
2D-Nav.	1	0.2332	0.2354	0.9999	0.9999	0.6172	2.0640	−68.63 ± 43.05	8.91 ± 15.14
	2	0.0410	0.8035	0.9999	0.9999	0.7111	0.0683	9.68 ± 26.49	121.25 ± 60.75
	3	0.3574	0.7163	0.9999	0.9999	0.7684	0.0096	125.15 ± 50.85	207.74 ± 32.35
	4	0.6639	0.6878	0.9999	0.9999	0.8379	0.0007	221.73 ± 22.02	207.0 5 ± 54.74
	10	0.1448	0.1485	0.9999	0.9999	0.5733	0.00063	224.31 ± 13.85	207.15 ± 26.38
	50	0.2263	0.1990	0.9999	0.9999	0.6063	0.00064	220.05±18.82	213.31 ± 21.43
	100	0.2062	0.2997	0.9999	0.9999	0.6265	0.00062	217.76 ± 20.79	226.95 ± 5.66
	200	0.3303	0.4587	0.9999	0.9999	0.6973	0.00066	217.43 ± 22.76	227.30 ± 5.38
	1000	0.4258	0.4311	0.9999	0.9999	0.7142	0.00086	212.72±21.68	225.87 ± 8.01

shows the $GTC$, $GT{C}_{mean}$, $\mathit{KNN}-\mathit{MSE}$ and associate mean reward performance of RL methods (DQN and PPO) per episode (average on 100 episodes) after $1e6$ steps for the mobile robot navigation tasks. When the environment and the task are simple, as shown in Figure 7 and Figure 8, a small state dimension could help the RL agent learn faster generally; however, when the agent interacts with a complicated environment in which the task is relatively difficult, the state dimension for the SRL model must be large enough to solve the task efficiently. Increasing the state dimension above a certain threshold, on the other hand, has no effect (positively or negatively) on RL performance.

5.4. Sample Quality

How does the sample quality affect RL algorithm performance?

The quality of samples used for SRL plays an important role in state representation especially in the visual-based RL. We usually need to do a trade-off between the computation complexity and the state representation accuracy, because visual-based RL demands more computation resource to process pixel-based data in order to learn effective state representation and policy, and a limited numbers of samples cannot cover the whole state space.

In this part, we generated five pixel datasets for mobile robot 1D navigation task and mobile robot one-random-target task separately, which were created by random policy, pre-trained PPO policy after 1000 (1e3) steps, pre-trained PPO policy after 10,000 (1e4) steps, pre-trained PPO policy after 100,000 (1e5) steps and pre-trained PPO policy after 1,000,000 (1e6) steps respectively. We used 10,000 samples from each kind of datasets to learn state representations. Reward is sparse (as illustrated in Table 2) and actions are discrete (encoded as integers) for all datasets.

Results:

Table 5. The influence of sample quality in mobile robot environment.

Task	Sampling Method	Correlation					KNN-MSE	Average Total Reward  PPO (1e6 Timesteps)
		Max Correlation Matrix				Mean
1D-Nav.	Random	0.8563	3.2195e-15	None	None	0.4282	0	238.35 ± 4.68
	PPO 1e3	0.8635	3.9844e-15	None	None	0.4318	0	238.45 ± 3.72
	PPO 1e4	0.6170	4.2631e-15	None	None	0.3085	1e-05	238.60 ± 4.11
	PPO 1e5	0.6170	4.2631e-15	None	None	0.3085	0	238.39 ± 3.66
	PPO 1e6	0.8110	2.5734e-15	None	None	0.4055	0	238.41 ± 4.06
2D-Nav.	Random	0.3303	0.4587	0.9999	0.9999	0.6973	0.00066	227.30 ± 5.38
	PPO 1e3	0.7826	0.3982	0.9999	0.9999	0.7952	0.00066	222.52 ± 13.86
	PPO 1e4	0.7804	0.3324	0.9999	0.9999	0.7782	0.00058	227.56 ± 5.25
	PPO 1e5	0.7804	0.3324	0.9999	0.9999	0.7782	0.00058	227.15 ± 5.96
	PPO 1e6	0.7784	0.3052	0.9999	0.9999	0.7704	0.00055	227.27 ± 8.48

gives the $GTC$ metrics, KNN-MSE metrics for several approaches and the associated RL performance using PPO algorithm. The corresponding RL performance curves are shown in Figure 9. We find that in the two mobile robot navigation tasks, the effects of sample quality are not consistent across environments. Datasets generated from random policy, on the other hand shows a relatively stable performance because the random policy allows the agent to explore more state and action space, effectively covering the huge policy search space.

5.5. Regularization Method

Is regularization item useful for learning SRL model?

One of the most common problems data science professionals face is to avoid overfitting. Regularization is a technique that makes slight modifications to the learning algorithm such that the model generalizes better. Thus, it improves the model’s performance on the unseen data as well.

In this section, we employ the regularized state representation ($ell1$ norm and $ell2$ norm regularization terms on SRL neural networks weight) to learn navigation strategies in the mobile robot navigation with random target task. We contrast the performance of different regularization methods and analyze the results.

${\ell }_1$ norm and ${\ell }_2$ norm are the most common types of regularization. ${\ell }_2$ norm is also known as weight decay as it forces the weights to decay towards zero (but not exactly zero); In ${\ell }_1$ norm, we penalize the absolute value of the weights. Unlike ${\ell }_2$, the weights may be reduced to zero here. Hence, it is very useful when we are trying to compress our model. Otherwise, we usually prefer ${\ell }_2$ over it. These update the general cost function by adding another term known as the regularization term.

$$\mathcal{J}=L(x,\widehat{x})+\lambda R\left(w\right),$$

(17)

where L is the regular loss function, R is the regularization term, and $\lambda$ is the scaling parameter in front of the regularization term to adjust the trade-off between the two objectives.

Due to the addition of this regularization item, the values of weight matrices decrease because it assumes that a neural network with smaller weight matrices leads to simpler models. Therefore, it will also reduce overfitting to quite an extent.

Results: In the mobile robot one-random-target task, we test several regularization scaling parameter $\lambda$ for ${\ell }_1$ norm and ${\ell }_2$ norm from ${10}^{-1}$ to ${10}^{-6}$ and choose the best one, which is ${10}^{-6}$ for ${\ell }_1$ norm and ${10}^{-3}$ for ${\ell }_2$ norm. As it shows in Figure 10, ${\ell }_2$ regularized SRL is second to ground truth, while ${\ell }_1$ is the worst. ${\ell }_1$ regularization in SRL can generate adequate sparsity. In comparison, ${\ell }_2$ regularization, which is easily solved by the ridge regression, is utilized to achieve faster and robust learning. ${\ell }_1$ and ${\ell }_2$ regularizations have their contributions for robust sparse representation. ${\ell }_1$ regularization is the preferred choice when having a high number of features as it provides sparse solutions. Although ${\ell }_1$ regularization technique has a computational benefit since zero-coefficient features can be avoided, it is discovered that ${\ell }_1$-norm based SRL is not necessary to produce positive results as expected.

5.6. State Representation Learning Methods

How can the SRL methods affect the RL algorithm performance?

As illustrated in Section 2.2, here we compare the performance of several SRL methods on the mobile robot one-random-target task, including forward model, inverse model, autoencoders, robotic priors, reward prediction, combination model and supervised model (state representations trained with GT states model).

Results:Figure 11 shows the average reward during the learning of PPO algorithm based on different SRL models. The effect of the forward model is the worst. The second last one is the autoencoder and reward prediction. As illustrated in Section 2.3, combining objectives make it possible to share the strengths of each model. In contrast it should be noted that the combination SRL model is not the most effective model. This is because that combing different objectives into a single embedding could sometimes result in the collision between themself, thus weakening the contribution of each component SRL model.

5.7. Reinforcement Learning Method

How do the state representations affect different RL algorithms’ performance?

In this section, we contrast several RL algorithms including both value-based algorithms (DQN) and policy-based algorithms (A2C, PPO, TRPO, ACKTR) on both ground truth state and the learned state.

Results: In the mobile robot navigation with random target task, as illustrated in Figure 12, RL algorithms converge faster and perform more consistently when they deal with with ground truth state in general. However, when we replace it with the learned state, the agent learns much slower and unstably, where DQN and TRPO perform worst. We observed that PPO was one of the best RL algorithm for SRL benchmarking. It allows us to obtain good performance and is consistent without needing to change any hyperparameters.

6. Summary Analysis and Discussion

Given the above results, we try to analyze whether representations show consistent performance in different environments. Regarding the given property measures, some representations have consistent performance while others do not. Some measures that do not have any strong relationship with the RL performance. Below we list major points summarized from the experiments.

• We found that auxiliary tasks do help with reinforcement learning during our experiments, but not all of them. On the contrary, adding a decoder for reconstructing the observation image is consistently bad at predicting expert knowledge such as the coordinate of the agent. Although the input-decoder auxiliary task has been empirically shown effective when it is used to prevent the representation from converging to 0, adding the decoder forces the agent to include every detail in the observation, even part of them is useless. Thus, one guess is that such a constraint requires a larger capacity in the representation than necessary and hurts the representation’s ability to extract useful information.
• $GT{C}_{\mathrm{mean}}$ is a good indicator of the performance that can be obtained in RL. When measuring $GT{C}_{\mathrm{mean}}$, we see that representations with a relatively high $GT{C}_{mean}$ score always show a good performance in RL (a higher mean reward).
• Auxiliary tasks generally improve the representation in terms of complexity reduction and decorrelation, regardless of whether the RL performance is improved or not. But there does not appear to be a clear relationship between the measures of these properties and the RL performance. In short, auxiliary tasks should be designed according to the task characteristics.
• Learning policies from vision-based DRL is sensitive to hyperparameter changes. For instance, hyperparameters (such as the embedded dimension) tuning of DQN is required in order to have decent results from the pixels (Figure 7 and Figure 8). However, the DRL performance is relatively stable by means of SRL methods. This can be explained by the reduced search space: the task is simpler to solve when features are already extracted.

7. Conclusions and Future Work

This work presents the advantages of decoupling feature extraction from policy learning in vision-based RL, on a set of robotics tasks. This decomposition reduces the search space, accelerates training, does not degrade final performances and gives more easily interpretable representations. Various experiments have been conducted to analyze the effect of several influence factors of the SRL model. We illustrate the variability in the reported methods and suggest guidelines to make future studies in SRL more reproducible and stable.

Although we have made conclusions based on plenty of experimental results, there remain things we would like to investigate further. The first thing is to improve the criterion and look for better definitions for properties listed in this work. Furthermore, there has been a substantial effort to characterize transfer, generalization, and overfitting in DRL, primarily in terms of performance [57,58,59]. This motivates the need for new domains and evaluation methodologies in SRL. Last but not least, future work should confirm the results in this paper by experimenting with real robots in more complex tasks.