Reproduce “3D Primitives are a Spatial Language for VLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junze Liu, Kun Qian, Florian Dubost, Kai Zhong, Arvind Srinivasan, Nan Chen, Anping Wang, Sam Zhang and 3 more
Researchers at Unity Technologies investigate a paradox where Vision-Language Models (VLMs) can generate executable 3D scene reconstruction code but struggle with direct natural language spatial questions about the same image. The work establishes 3D geometric primitives as a "spatial language" for VLMs and proposes Code-CoT and S M -FT methods to enhance spatial reasoning, showing up to +17% improvement on real-photo benchmarks.
Key insights
- VLM spatial knowledge is not uniformly accessible but is significantly entangled with the model's familiarity and prior training on specific scene-code languages, leading to up to 5.7 times performance variance.
- 3D geometric primitives, when expressed in executable code, function as a compositional spatial vocabulary for VLMs, providing a structured scaffold for reasoning that can be leveraged at inference or training time.
- Self-supervised distillation of primitive spatial knowledge through S M -FT can generalize effectively to out-of-distribution real-world images, suggesting a transferrable spatial understanding without requiring human labels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.