Reproduce “Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Enshen Zhou, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Mengzhen Liu, Yi Han, Yuheng Ji and 7 more
RoboTracer introduces a Vision-Language Model framework that enables embodied robots to interpret complex, spatially constrained instructions by generating 3D spatial traces. It achieves this by integrating 3D spatial referring and measuring capabilities through a novel architecture and metric-sensitive reinforcement fine-tuning, demonstrating robust and generalized performance in real-world robotic tasks.
Key insights
- A decoupled (u,v,d) point formulation for 3D spatial traces simplifies VLM training, improves accuracy, and supports co-training with diverse 2D/3D datasets.
- Integrating explicit geometric cues through a universal spatial encoder and supervising a scale decoder significantly enhances 3D metric awareness and reasoning capabilities within VLMs.
- Reinforcement Fine-tuning (RFT) with metric-sensitive process rewards, which supervise intermediate perceptual steps, is crucial for robust multi-step, metric-grounded reasoning in 3D environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.