Hybrid Bipedal Locomotion Based on Reinforcement Learning and Heuristics

Abstract

Locomotion control has long been vital to legged robots. Agile locomotion can be implemented through either model-based controller or reinforcement learning. It is proven that robust controllers can be obtained through model-based methods and learning-based policies have advantages in generalization. This paper proposed a hybrid framework of locomotion controller that combines deep reinforcement learning and simple heuristic policy and assigns them to different activation phases, which provides guidance for adaptive training without producing conflicts between heuristic knowledge and learned policies. The training in simulation follows a step-by-step stochastic curriculum to guarantee success. Domain randomization during training and assistive extra feedback loops on real robot are also adopted to smooth the transition to the real world. Comparison experiments are carried out on both simulated and real Wukong-IV humanoid robots, and the proposed hybrid approach matches the canonical end-to-end approaches with higher rate of success, faster converging speed, and 60% less tracking error in velocity tracking tasks.

1. Introduction

In recent years, various legged robots are developed for traversing through rough and dangerous terrains and working as a replacement for human effort [1]. Specially, dynamic locomotion control, as an essential part of a robust and agile legged robot, has been studied from various aspects.

Model-based methods are first studied and developed [2]. Heuristics methods are widely adopted during early exploration, Raibert proposed hopping controller [2] and extended to bipedal and quadruple robots, and Pratt proposed Virtual Actuator Control [3] and Virtual Model Control (VMC) [4] on bipedal locomotion tasks. These methods adopt reduced modeling and abstract the legged robot into a linear inverted pendulum (LIP) [5] and its derivatives. To further improve the decision, optimization-based methods are introduced to locomotion control [6]. To prevent redundant iterations when searching for the optimal solution, Bledt et al. proposed a regularization-based approach [7]. It uses sub-optimal heuristics to inform the solver through cost function. As a low-cost policy that can approximate optimal solutions with linearization around the working limit cycle, heuristics are still valuable.

Model-based methods have made great advances and are tested to be precise, predictive, and adaptive in various scenes. As they are all based on simplified models that guarantee convergence and simplicity, the accuracy and performance drop when being far away from the working point.

As an emerging and model-free technology, deep reinforcement learning (DRL) has been a popular research field. Previous studies have demonstrated its potential for locomotion tasks. Haarnoja and Tan [8,9] first utilized learning-based gaits on a real robot and verified the feasibility of the end-to-end route. Subsequently, the work in [10,11,12] presents learning separate skills such as trotting and fall recovery using a similar framework. As for bipedal locomotion, a harder task, Hurst [13,14,15] proposed special periodical switching rewards and achieved robust locomotion over planes and stairs. To recombine multiple motor skills into an adaptive multi-modal policy within one end-to-end framework, Yang [16] designed a switching mechanism guided by a phase indicator. Based on that, the multi-expert learning architecture [17] was proposed. It fuses multiple NNs into one according to a gating network’s output to produce adaptive behaviors in response to changing situations. Meanwhile, expanding information sources with perceptual sensing such as heightmap scanning [18] and introducing latent encoders are also means to enhance trained policies. ANYmal robot conquered a series of challenging terrains with privilege learning and latent encoder [19,20]. Kumar et al. proposed Rapid Motor Adaptation framework [21] that enables latent space identification and realized quadrupedal locomotion without predefined references or trajectory generators.

On basis of end-to-end frameworks where the action is completely learned, the sum of model-based control signals and the action of learned policies provide external guidance and avoid pointless blind exploration during training. Low-cost model-based controllers such as central pattern generator [19,22,23], model-based gait library [24,25,26], and heuristic references [27,28] are often adopted in these approaches to achieve agile real-time controlled locomotion. In these frameworks, NN policies learn the residual between optimal decision and reference given by model-based modules. A common issue facing such a summation mechanism is that learned policy might conflict with the other component and cause rigid policy constrained by reference, or the learned policy overwrites the other [22]. Both parts counteract, resulting in a mediocre final performance.

The most similar methodology to ours is linear policy locomotion developed by Krishna [29,30], where the biped’s swing leg is controlled by simple NN policy, and the stance leg is controlled by a model-based controller. However, we have an opposite understanding of the traits of walking phases. In previous studies, we observed that the stance leg performs agile movements through acting ground reaction force (GRF) instead of acting specific joint position trajectory. The swing leg works with a relatively low payload and without restriction from the ground, but the foothold decided by it significantly impacts the subsequent body states. Thus, joint-level accuracy and higher stiffness of swinging control are required. Swing control is a rather deterministic tracking task and can be accomplished by heuristic method and similar control policies with fixed-based manipulators [31]. Meanwhile, the stance leg faces more uncertainty during interaction with terrain and is often controlled by torque controllers [25]. Adopting a learning-based method on stance control can improve robustness compared with model-based approaches. It also narrows down the space for exploration and grants higher efficiency in training.

In pursuit of better performance, separated controllers should be activated during the corresponding phase and adapt to the phase’s attribute, instead of applying a unified controller to fulfill different demands. Inspired by such philosophy, the paper proposes a novel conflict-free hybrid approach to generate adaptive bipedal locomotion with a higher success rate and is easy to transplant to other biped robots with different joint configurations since the policy produces GRF commands. The proposed method is computationally efficient, and also light-weight since there is no iteration-based numerical solver involved in the framework. The main contributions of this paper are as follows:

• We proposed an efficient hybrid locomotion policy that divides the task into a stance part using model-free learning-based stance control and a swing part using heuristics swing control. The trained policy can perform controllable and regulated gait.
• To prevent policy divergence caused by fierce randomization at early stages, we proposed a parallel curriculum learning schema with two-stage progressive domain randomization for challenging locomotion tasks.
• To further reduce the magnitude of sensor randomization which withdraws training speed, we proposed heuristic-based regularization feedback loops on real-world robot systems in addition to a two-branch hybrid controller for resisting simulation model mismatch and assisting stabilization.

The organization of this paper is as follows: Section 2 describes the hardware and software platform and notations. Section 3 introduces the overall structure of our method. Section 4 presents simulation and deployment results and comparisons with an end-to-end benchmark. Finally, Section 5 states the discussion and inspiration for future work.

2. Platform and Notations

2.1. Robot Model

The robot platform adopted in our work is Wukong-IV adult-size humanoid. Wukong-IV humanoid is designed and built by our research team. It is 1.4 m tall and weighs 45 kg, actuated by electric motor joints. The robot has 6 degrees of freedom (DoF) on each leg and 4 DoFs on each arm. As a bionic humanoid robot, Wukong-IV’s joint configuration and mass distribution bare resemblance to biological humans. A picture of its appearance is shown in Figure 1A, the articulated system model used in simulation training is shown in Figure 1B, and more detailed specifications are listed in

Table A1. Wukong-IV Hardware Specifications.

Parameters	Value
Mass	45 kg
Height	1.4 m
Thigh Length	0.3 m
Shank Length	0.3 m
DoF per Leg	6
DoF per Arm	7
Power Source	Lithium Battery
Joint Actuator	Brushless Motor

.

The simulation environment is built with RaiSim [32]. Policy neural networks and their training algorithm are utilized with PyTorch [33]. All training is performed on a desktop workstation with a central processor of Intel Xeon Gold 6242R and a graphic processor of NVIDIA Geforce RTX 3090.

2.2. Math Notations

In this article, the state of WuKong-IV’s floating base is denoted by $q_b$ = {$p_x$, $p_y$, $p_z$, $\psi$, $\theta$, $\varphi$}. The first three elements compose base position, also denoted by p. The last three are roll, pitch, and yaw angle of base. Robot base quaternion is denoted by $\xi$. Base linear and angular velocity is v and $\omega$. The joint level state is composed of joint angle q and joint angle velocity $\dot{q}$. In stance control, the policy gets input $o_t$ and outputs action $a_t$. The swing foot’s position in the robot frame is denoted by e. Joint torques of legs are denoted by ${\tau }_{stance}$ and ${\tau }_{swing}$. To represents which foot touches the ground, gait is denoted by c.

For all notations, ${(·)}_t$ means the quantity in time step t, ${(·)}^{*}$ means the expected value of the quantity, and $\delta (·)$ means compensation to be added to the scripted quantity.

3. Materials and Methods

The control frame of hybrid locomotion policy can be described in Figure 2. There are three main branches in the overall structure. The middle branch (blue blocks) shows the learning-based stance control policy (Section 3.1). The network outputs the expected GRF to the stance leg. GRF maps to joint torque via jacobian and then is acted by the robot. The upper branch (orange blocks) shows the heuristic swinging control policy (Section 3.3) which is a position-based method inspired by capture point method. The next foot placement of swing legs is calculated from a clock signal and robot state, then cosine interpolation generates the air position of swing foot, then inverse kinematics converts the position into joint position, which is tracked by low-level PD. The lower branch (red block) is only used in physical deployment (Section 3.5).

3.1. Learning-Based Stance Control

3.1.1. Observation Space and Action Space

Observation space $o_t$ consists of 33 dimensions of proprioceptive information. The observation vectors are normalized before being given to policy to neutralize the amplitude difference between various observation channels (For normalization parameter update strategy, see Section 3.1.4). Its components are as follows.

$$o_t=\{q_{b,t},q_b^{*},{\omega }_t,v_t,{\omega }^{*},v^{*},e_t,a_{t-1}\}$$

(1)

Action space $a_t$ consists of 6-dimension expected GRF which includes reaction force F and torque $\tau$ in XYZ directions. $F_y$ component of action is mirrored along zero at the left stance phase so that the policy network only has to explore a simpler region rather than two unconnected regions. Its component is as follows.

$$a_t=\{F_x,F_y,F_z,{\tau }_x,{\tau }_y,{\tau }_z\}$$

(2)

3.1.2. Reward Design

Reward function design is also vital for training. Improper reward function would lead to reward hacking, a phenomenon where policy takes unintended behavior to obtain high reward value. Therefore, reward value should be a comprehensive evaluation of task performance and meanwhile dense and smooth enough to learn. For locomotion tasks, the reward function mainly consists of tracking terms that follow a goal and stabilizing terms that prevent falling or collision. In our work, the reward function is composed of three objectives including balancing, command tracking, and gait regulating. The full reward composition is shown below.

$$\begin{array}{ccc}\hfill R_{track}& =& F_{{\alpha }_1,{\beta }_1}\left(\right||v_t^{*}-v_t\left|\right|)\hfill \end{array}$$

(3)

$$\begin{array}{ccc}\hfill R_{pose}& =& F_{{\alpha }_2,{\beta }_2}(1-{\xi }_t^{*T}·{\xi }_t)\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill R_{height}& =& F_{{\alpha }_3,{\beta }_3}\left(\right|p_{z,t}^{*}-p_{z,t}\left|\right)\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill R_{gait}& =& F_{{\alpha }_4,{\beta }_4}\left(\right|c_t^{*}-c_t\left|\right)\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill R_{slip}& =& F_{{\alpha }_5,{\beta }_5}\left(\right||v_{ankle,t}\left|\right|)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill F_{\alpha ,\beta }\left(x\right)& =& \alpha e^{-\beta x^2}\hfill \end{array}$$

(8)

$$R_{fall}=\left\{\begin{array}{ccc}\hfill K& \mathrm{for}& \mathrm{falling}\hfill \\ \hfill 0& \mathrm{for}& \mathrm{other}\hfill \end{array}\right.$$

(9)

$$R_{sum}=R_{track}+R_{pose}+R_{height}+R_{gait}+R_{slip}+R_{fall}$$

(10)

where $v_{ankle,t}$ means the ankle velocity of the stance leg. $R_{pose}$ and $R_{slip}$ require the robot to maintain balance with firm steps, $R_{track}$ and $R_{height}$ encourage the robot to traverse under certain commands, and $R_{gait}$ regulates the gait frequency to meet the expected one. All vectors above are expressed in the world frame. $R_{fall}$ penalizes major failures. $F_{\alpha ,\beta }\left(x\right)$ is the Gaussian kernel function, and subscripts $\alpha$ and $\beta$ are weight coefficients that need to be tuned. Every term except $R_{fall}$ is transformed from negative errors to positive rewards with a Gaussian kernel to prevent proactive termination and set up upper boundaries for each reward term to prevent any term from over-growing and shadowing the others. For specific parameters we used in our experiments, check

Table A2. Reward design and parameters.

Reward Term	Coefficient
$R_{track}$	${\alpha }_1$ = 0.3, ${\beta }_1$ = 40
$R_{pose}$	${\alpha }_2$ = 0.3, ${\beta }_2$ = 500
$R_{height}$	${\alpha }_3$ = 0.3, ${\beta }_3$ = 400
$R_{gait}$	${\alpha }_4$ = 0.1, ${\beta }_4$ = 10
$R_{slip}$	${\alpha }_5$ = 0.3, ${\beta }_5$ = 100

For R_fall, K = −600

.

3.1.3. Policy Representation

In the aforementioned studies [8,9,10,11,12,14,15,19,20,21,26], Actor-Critic framework [34] is adopted to train the policy for locomotion tasks. Multiple layer perception (MLP) with two hidden layers and tanh as activation function is chosen for policy representation in our study because of its simple structure, which boosts training and transition to non-Python machines. Actor and critic networks share the same structure, configuration, and observation input. The policy is trained with the widely-used Proximal Policy Optimization algorithm (PPO) [35] based on surrogate loss, importance sampling, and AdamW optimizer. No experience replay or any buffer of such kind is used in our framework, and all data collected from simulation would be trained only once before being flushed away. The combination of PPO and MLP has already been proved to be feasible in learning robot locomotion policies by multiple research works [11,12,19,20]. Advanced network structures such as LSTM or Transformer can also fit into the proposed framework as policy representation, but more tuning and training effort will be required. The learning rate and its decay threshold have been optimized to accelerate early convergence and avoid over-fitting. Other hyper-parameters are maintained as default values provided by OpenAI Baselines [36]. The detailed hyper-parameters can be found in

Table 1. Neural Network Hyper Parameters

Parameters	Value
Network Type	MLP
Latent Layer	2
Latent Node Number	256
Activation	Tanh

and

Table A3. Training Hyper Parameters.

Parameter	Value
Parallel Environment	150
Maximum time steps per epoch (Stage One)	2000
Learning Rate	1 × 10${}^{-4}$
Discount Factor	0.996
Curriculum Reward Threshold	2300

.

3.1.4. Adaptive Normalization

Adaptive normalization is applied before the network (the blue block in Figure 2), where the normalization parameters are updated every epoch according to observation storage as shown in equations below and finally converge to the desired values associated with the task so that the system designers do not have to re-tune the normalization module when changing task.

$$\begin{array}{ccc}\hfill N_i& =& N_{i-1}+n_i\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\mu }_i& =& {\mu }_{i-1}+({\overline{s}}_i-{\mu }_{i-1})\frac{n_i}{N_i}\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill {\sigma }_i^2& =& \frac{1}{N_i}[N_{i-1}{\sigma }_{i-1}^2+n_i{v}_i+N_{i-1}\sqrt{\frac{n_i}{N_i}({\mu }_i-{\mu }_{i-1})}]\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill s^{\prime }& =& \frac{s-{\mu }_i}{{\sigma }_i^2+{10}^{-8}},s\in S_i\hfill \end{array}$$

(14)

where $s^{\prime }$ is normalized observation vector. $S_i$ is the ith batch retrieved from the relevant training episode. $n_i$ is the number of samples in batch $S_i$. $N_i$ is the total number of samples. ${\overline{s}}_i$ and $v_i$ are the mean value and sample variance of $S_i$. ${\mu }_i$ and ${\sigma }_i^2$ are the mean value and variance for normalization after the ith update.

3.2. Curriculum Schema

To improve adaption to various velocity commands and robustness against noisy observations, the policies are trained with a two-stage parallel curriculum as shown in Figure 3.

Before training, parallel environments are built for both accelerating data collection and fostering adaptability. Dynamics randomization is applied to every environment upon setup and remains constant during training. In this way, the policy is exposed to and forced to adapt to a wide range of possible dynamics in training, which reduces the reality gap resulting from dynamics mismatch. Random selection is called in each environment when resetting occurs. It randomly selects a target velocity under world frame from a discrete set. This method is found to be more effective than directly generating continuous random target velocity at every time step. Distribution of random velocity target can be expressed as:

$$P(v^{*}=i)=\frac{i}{2}+0.1,i\in \{0.0,0.2,0.4,0.6\}$$

(15)

In the first stage, the policy is trained in an ideal environment and adaptively updates normalization parameters ${\mu }_i$ and ${\sigma }_i^2$. The policy network receives raw observation without any external disturbance. Stage one ends when the average overall reward per trajectory reaches a threshold that guarantees velocity tracking without falling.

In the second stage, we add domain randomization to simulate noisy sensors and external force perturbation on the torso. In test runs of some stage one results, instant failure might happen after epoch time ends. Simulation time per epoch per environment is prolonged by half in this stage to prevent this problem. Normalization parameters become fixed in this stage, in case they are violated by randomization and new failure at the beginning of the new stage.

Detailed parameter configurations are listed in

Table A4. Randomization Parameters.

Variable	Distribution
Link Mass	$\Delta m\sim U (0.85m,1.15m)$
Link CoM Position	$\Delta x\sim U (0.9x,1.1x)$
Link Inertia	$\Delta I\sim U (0.8I,1.2I)$
Sensor	$\Delta x\sim U (0.95x,1.05x)$
External Force along X Axis	$f_x\sim U (-10\mathtt{N},+10\mathtt{N})$
External Force along Y Axis	$f_y\sim U (-10\mathtt{N},+10\mathtt{N})$

Random external force is activated for 0.5 s in every 2.5 s.

.

3.3. Heuristic Swinging Control

Despite its limitations, heuristics are simple and intuitive enough to be widely used. Inspired by capture point method [37], we adopt a simple linear policy for foot placement planning. The heuristic foot placement policy can be described as follows:

$$e_t=\delta e_t+k_1·v_t+k_2·v_t^{*}$$

(16)

where e is step displacement relative to hip expressed in world frame. $\delta e$ is an offset vector. $k_1$ and $k_2$ are both coefficient vectors to be tuned manually.

Foot placement under world frame then has to be converted into joint command. The swing leg is not constrained by contact forces, thus position-based control method can handle it. To avoid collision and singularity, we use trigonometric interpolation in Cartesian space as described below, and transform it into joint space commands via inverse kinematics. We adopted a geometric inverse kinematics computation The planning process is also shown in Figure 4. Joint level position and velocity targets are sent to the low-level controller to be tracked.

$$p_w\left({\varphi }_t\right)=\left\{\begin{array}{ccc}\hfill A_{1}cos2\pi {\varphi }_t+B_1& \mathrm{for}& 0.0\le {\varphi }_t<0.5\hfill \\ \hfill A_{2}cos2\pi {\varphi }_t+B_2& \mathrm{for}& 0.5\le {\varphi }_t<1.0\hfill \end{array}\right.$$

(17)

$$\begin{array}{ccc}\hfill A_1& =& 0.5\times (z_{begin}-z_{mid})\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill A_2& =& 0.5\times (z_{mid}-z_{end})\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill B_1& =& 0.5\times (z_{begin}+z_{mid})\hfill \end{array}$$

(20)

$$\begin{array}{ccc}\hfill B_2& =& 0.5\times (z_{mid}+z_{end})\hfill \end{array}$$

(21)

where $z_{begin}$, $z_{end}$, and $z_{mid}$ refer to the z-axis positions of the swing foot when the swing phase begins, ends, and is in the middle. All positions in the equations are expressed in world frame. $\varphi$ refers to the phase value which increases from 0.0 to 1.0 in a gait period.

Phase switching is also done with heuristic conditions. When both the estimated contact force on the swing foot and the time after the last switch reaches their thresholds, or time reaches a larger threshold, a new switch is triggered.

3.4. Low-Level Controller

Different low-level controllers are placed after mentioned policies to convert their output signals to joint-level torque commands. For the stance leg, when performing stable locomotion, the acceleration of the robot is trivial enough to be ignored, so that the expected GRF can be approximately equal to the opposite of the end-effect force applied by the stance foot. For the swing leg, a simple proportional-derivative controller calculates feedback joint torque commands. The formulas are defined as follows:

$$\begin{array}{ccc}\hfill {\tau }_{stance}& =& J_{stance}^T(-a_t)\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill {\tau }_{swing}& =& K_p·(q_t^{*}-q_t)+K_d·({\dot{q_t}}^{*}-\dot{q_t})\hfill \end{array}$$

(23)

where $J_{stance}^T$ is the force jacobian of stance leg, $K_p$ and $K_d$ are coefficient vectors to be tuned.

3.5. Regularized Sim-to-Real Transfer

In addition to randomization mentioned in Section 3.2, dynamics randomization also exists throughout all training stages. Randomized parameters including mass, inertia, and CoM position of links, are generated and fixed when parallel environments are built. See detailed distribution in Table A4.

To maintain the performance of the trained policy, we deploy it with heuristic-based feedback loops to provide regularization on real robots to assist in stabilization around the desired state. Inspired by VMC and Regularized Predictive Control methods, we condition compensation for low-level controllers with floating-base state errors based on heuristic control laws. The feedback formulations are as shown below.

$$\begin{array}{ccc}\hfill \Delta F_z^{*}& =& K_z(p_z^{*}-p_z)\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill \Delta {\tau }_{abduction}& =& K_y({\theta }_t^{*}-{\theta }_t)\hfill \end{array}$$

(25)

where $K_y$ and $K_z$ are both coefficients. $F_z$ is expected GRF along Z axis, and $\theta$ is the roll angle of the robot base.

4. Results

4.1. Simulation Training Results

Figure 5 compares the performance of our proposed method with other learning approaches. Shaded areas refer to the range of a group of curves, and the solid curve refers to the average of the group. All approaches share the same hyper-parameters. As for heuristics and low-level parameters, we roughly tuned them to a near-optimal state that would not cause frequent failure and proactive termination behaviors in the first 50 training epochs. We chose the prior-free end-to-end approach proposed by Hurst [14] as the canonical baseline, and an environment that enables domain randomization from the beginning to demonstrate the practicability of the curriculum algorithm.

The proposed hybrid method earns higher rewards on average than the other approaches in both stages. In the early stage of the proposed training method, the curves have large variance because, in each test run, the curriculum stage switches according to its reward level, and the newly introduced disturbance in the second stage causes more failures and reward drop.

In contrast, using end-to-end baseline approach the policy might occasionally perform as nicely as the converged state of the proposed method, but the overall performance is found to be unstable and bares a lower chance of success in practice.

The results of the policy without the curriculum are shown in orange. Corresponding curves reveal relatively weak performance with a high risk of falling. The possible reason is that an over-challenging environment at the beginning leads to frequent failure, then lack of valid data to update policy and normalization parameters, and eventually early convergence before the policy learns to maintain balance.

Figure 6 shows the tracking performances of horizontal velocity. In the 40 s test episode, the velocity reference is composed of a series of step signals. Compared with end-to-end training results, the hybrid policy is proved able to track velocity reference up to 2 km/h with less tracking error. The mean squared error of the hybrid policy and end-to-end baseline policy are $2.232\times {10}^{-3}$ and $1.282\times {10}^{-2}$. We also tested the same trained hybrid policy using robot models with 5 kg heavier torso (purple curve) and 5 kg lighter torso (red curve). Generally, although payload variations cause larger oscillations and slower responding performance, the proposed hybrid policy shows resistance to mass and inertia discrepancy and finishes tracking tasks without falling.

Figure 7 shows the phase plane plot of X velocity tracking error using different policies. All curves contain data of 2 s since velocity references shift. The curves of the hybrid policy (orange and red) display a spiral pattern attracted to a region centering (0, 0). The stability of the proposed hybrid method is displayed. The end-to-end baseline policy converges to an offset region with a relatively random pattern, which suggests larger tracking error and less stability.

Figure 8 shows the comparison of contact points and joint-level states. Trained hybrid policy produces a regular gait with a frequency of 2.1 Hz, and steady contact indicates firm contact within the stance phase. Compared with the hybrid policy, the baseline policy’s performance is less periodic for its spiky joint angle curves and fitful contacts during the left stance phase.

Figure 9 displays how the robot controlled by hybrid policy regains its balance when being externally pushed. A lateral pushing force of 14 N in world frame is randomly applied on a point on its torso for 0.5 s when the robot is walking 0.6 m/s. After the pose is dragged away from the upright state, two horizontal steps are then taken by the heuristic policy, meantime learned policy maintains the posture of the torso. It is proven that the trained hybrid locomotion controller has enough robustness to endure pushing force and retain balance. In contrast, end-to-end baseline policy cannot endure the same pushing force and falls with brittle actions. In consideration of safety for humans and robots, we did not transfer trained baseline policy onto real robots.

4.2. Sim-to-Real Transfer

Figure 10 shows snapshots of a real Wukong-IV robot controlled with the trained hybrid policy. The robot can maintain balance and step in place. From Figure 11 it is clear that equipped with assistive feedback, the performance of the proposed hybrid policy on a real robot is more stable than ones without extra feedback loops. Unaided policies produce vigorous oscillations and lead to failure soon after.

To prove that assistive feedback is not overwriting the policy, in other experiments, the policy network was disabled and assistive feedback was in charge of actuating the robot. As expected, despite multiple fine-tuning being carried out, the robot still instantly fell and crashed.

5. Discussion

We have presented a hybrid locomotion policy to produce nice results with achieving resilient bipedal walking. Learning and heuristics cooperate within the proposed structure to finish the locomotion task. In the simulation, the policy is capable of tracking velocity command up to 0.6 m/s. Sim-to-real transfer to Wukong-IV is feasible with assistive feedback loops. The proposed hybrid framework yields higher reward and better performance than similar methods in terms of balance keeping and command tracking accuracy, and thus proved to be a stable and efficient approach to training a locomotion controller with adaption and robustness.

However, there are also limitations in the proposed methodology. Firstly, the performance of policies on a real robot is degraded compared with it in the simulation. The most possible reason is that the simulation and solution results of the software are too ideal, resulting in a large gap with reality. During deployment, policies cannot fully bridge the sim-to-real gap even with assistive feedback loops. In order to improve the performance of real robots, more advanced accurate identification work should be done, and detailed research on more oriented domain randomization combined with the dynamics of legged robots will be demanded in the future. Secondly, the proposed locomotion is still a blind one and just works on flat ground. For a humanoid robot, a highly nonlinear system, relying solely on proprioceptive control, is not conducive for the robot to face complex terrain. Future work is also recommended to adapt to more challenging terrain such as stairs and uneven surfaces with perceptual information.