Reproduce “Learning Visual Locomotion with Cross-Modal Supervision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Antonio Loquercio, Ashish Kumar, Jitendra Malik
Researchers at the University of California, Berkeley developed Cross-Modal Supervision (CMS), a self-supervised method for quadruped robots to learn visual locomotion using only a monocular RGB camera. The approach allowed a robot to train its visual perception module directly in the real world, achieving 100% success navigating stairs with 26% higher average speed compared to a blind policy, and demonstrating rapid adaptation to visual changes.
Key insights
- Proprioceptive feedback, although reactive, can be time-shifted to provide accurate 'ground truth' supervision for a visual look-ahead module, enabling self-supervised learning of vision in the real world.
- Decoupling the motor policy learning (in simulation) from the visual perception learning (in the real world via self-supervision) effectively addresses the sim-to-real gap for RGB images in locomotion.
- Even a moderate look-ahead distance (e.g., 15cm) significantly improves a robot's ability to proactively adapt its gait for complex terrains, enhancing speed and success rates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.