Reproduce “RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang and 3 more
RoboRefer predicts 2D image-plane points from instructions that combine spatial clues, targeting objects or placement locations. It uses separate RGB and depth pathways, first training spatial perception with supervised fine-tuning, then reinforcement fine-tuning with rewards for intermediate perceptions and final points. On RefSpatial-Bench, RoboRefer-2B-RFT averages 53.0% success versus Gemini-2.5-Pro at 35.6%; it predicts 2D points, not robot-ready 3D positions.
Key insights
- A dedicated depth encoder, separate from the RGB encoder, effectively prevents modality interference and enhances precise 3D spatial understanding without compromising the VLM's general visual knowledge.
- Reinforcement Fine-tuning (RFT) with metric-sensitive process reward functions is crucial for teaching explicit, generalized multi-step spatial reasoning, overcoming the memorization tendencies of SFT for complex, sequential tasks.
- A comprehensive data recipe, integrating 2D web images, 3D embodied videos, and simulated scenes with ground-truth reasoning processes, is essential for training models capable of both precise spatial understanding and robust multi-step reasoning across diverse environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.