Reproduce “GeoLoco: Leveraging 3D Geometric Priors from Visual Foundation Model for Robust RGB-Only Humanoid Locomotion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yufei Liu, Xieyuanli Chen, Hainan Pan, Chenghao Shi, Yanjie Chen, Kaihong Huang, Zhiwen Zeng, Huimin Lu
The GeoLoco framework enables robust humanoid locomotion using only a monocular RGB camera by extracting 3D geometric priors from a frozen Visual Foundation Model and fusing them with proprioception via cross-attention. This approach achieves reliable zero-shot sim-to-real transfer, demonstrating 80% success on 0.23m stairs and 70% on 0.25m gaps in real-world deployments.
Key insights
- Robust 3D geometric information can be effectively extracted from monocular RGB by leveraging the inherent geometric priors embedded within pre-trained Visual Foundation Models, offering a viable alternative to active depth sensors.
- A proprioceptive-query cross-attention mechanism effectively integrates high-dimensional visual features with reactive motor control, enabling dynamic modulation of visual focus based on the robot's instantaneous physical state.
- Auxiliary tasks, specifically terrain reconstruction and velocity estimation, are crucial for grounding high-dimensional visual representations in physical geometry, preventing overfitting to superficial visual textures and enhancing sim-to-real transferability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.