Reproduce “CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion” on OpenResearch

About this paper

Yuan Hao, Ruiqi Yu, Shixin Luo, Guoteng Zhang, Jun Wu, Qiuguo Zhu

Researchers at Zhejiang University developed CReF, a single-stage framework for depth-conditioned humanoid locomotion that directly maps proprioception and raw forward-facing depth to joint position targets. The policy demonstrated robust zero-shot transfer to a physical robot, achieving stable navigation on complex real-world terrains and reducing median absolute deviation of descending stair touchdown positions from 9.8 cm to 1.4 cm.

Key insights

• Robust humanoid locomotion can be achieved through a direct, single-stage policy learning from raw depth and proprioception, without requiring explicit geometric intermediates or multi-stage supervision.
• Proprioception-conditioned cross-modal attention is essential for adaptively extracting relevant terrain information from raw depth, particularly for navigating challenging and out-of-distribution environments.
• Adaptive fusion of multimodal and temporal features, enabled by GRF and a GRU with a highway gate, allows the policy to dynamically adjust its reliance on immediate observations versus historical context based on perceptual ambiguity or locomotion phase.
• An anticipatory, terrain-aware foothold placement reward effectively shapes foot contact behavior, significantly improving touchdown precision and repeatability on complex terrains like stairs.

Summary by alphaXiv.