Reproduce “Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers” on OpenResearch

About this paper

Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, Xiaolong Wang

Researchers from UC San Diego and Tsinghua University developed LocoTransformer, an end-to-end Reinforcement Learning framework that fuses proprioceptive and depth sensor data with a cross-modal Transformer. This approach enables quadrupedal robots to navigate complex environments, showing up to a 92% increase in distance moved and fewer collisions in real-world tasks compared to baseline methods.

Key insights

• The Transformer's cross-modal attention mechanism allows the policy to selectively attend to task-relevant visual regions (e.g., obstacles, goal), facilitating proactive planning and reactive adjustments.
• End-to-end learning with cross-modal Transformers yields superior generalization and robustness for sim-to-real transfer, capturing fundamental object structures rather than overfitting to simulation appearances.
• Naive concatenation of visual and proprioceptive features is less effective, as the Transformer's ability to model spatial and cross-modal dependencies is crucial for complex terrain navigation and obstacle avoidance.

Summary by alphaXiv.