Reproduce “Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie” on OpenResearch

About this paper

Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonathan Hurst, Michiel van de Panne

An iterative deep reinforcement learning framework enables flexible policy refinement and consolidation for human-scale bipedal robots. This approach allowed the Cassie robot to acquire diverse and robust locomotion skills, including variable-speed walking and high-stepping gaits, with direct simulation-to-real transfer without dynamics randomization.

Key insights

• DASS tuple collection, by sampling states from the stochastic execution of an expert policy and recording deterministic mean actions, provides robust training data that mitigates covariate shift in imitation learning.
• The combined RL and DASS-based supervised learning approach enables flexible and iterative policy refinement, allowing for targeted behavioral changes without causing catastrophic forgetting of previously acquired skills.
• Successful direct sim-to-real transfer on a human-scale bipedal robot (Cassie) without dynamics randomization is achievable, suggesting that factors like learning stochastic policies at a lower control rate contribute to inherent robustness.

Summary by alphaXiv.