Reproduce “Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Enshen Zhou, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Mengzhen Liu, Yi Han, Yuheng Ji and 7 more

RoboTracer introduces a Vision-Language Model framework that enables embodied robots to interpret complex, spatially constrained instructions by generating 3D spatial traces. It achieves this by integrating 3D spatial referring and measuring capabilities through a novel architecture and metric-sensitive reinforcement fine-tuning, demonstrating robust and generalized performance in real-world robotic tasks.

Key insights

  • A decoupled (u,v,d) point formulation for 3D spatial traces simplifies VLM training, improves accuracy, and supports co-training with diverse 2D/3D datasets.
  • Integrating explicit geometric cues through a universal spatial encoder and supervising a scale decoder significantly enhances 3D metric awareness and reasoning capabilities within VLMs.
  • Reinforcement Fine-tuning (RFT) with metric-sensitive process rewards, which supervise intermediate perceptual steps, is crucial for robust multi-step, metric-grounded reasoning in 3D environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.