Reproduce “Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning” on OpenResearch

About this paper

Feiyang Wu, Zhaoyuan Gu, Hanran Wu, Anqi Wu, Ye Zhao

A two-stage Inverse Reinforcement Learning (IRL) framework is introduced for bipedal robots, which learns free-form reward functions from demonstrations and interprets them to improve locomotion on uneven terrains. The method enhances learning speed and robustness, achieving higher average CoM sagittal velocities (e.g., 0.6249 m/s on perturbed terrain compared to 0.2617 m/s for baselines) and mitigating undesirable gaits by leveraging insights from reward analysis.

Key insights

• Applying IPMD within a Maximum Entropy and average-reward framework can effectively learn free-form reward functions for bipedal locomotion, potentially outperforming the initial expert demonstrations.
• Interpreting black-box neural network reward functions through VDN approximation and Integrated Gradients reveals crucial features (e.g., CoM lateral velocity, pelvis pitch, foot forces) that drive expert locomotion, providing explainability.
• Reward functions learned from flat-ground demonstrations can transfer knowledge to significantly enhance learning speed and robustness in novel, challenging terrains, and insights from reward analysis can be used to refine these rewards to prevent undesirable behaviors.

Summary by alphaXiv.