Reproduce “Zero-Shot 3D Visual Grounding from Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rong Li, Shijie Li, Lingdong Kong, Xulei Yang, Junwei Liang
Researchers from HKUST, A*STAR, and NUS developed SeeGround, a training-free framework for zero-shot 3D Visual Grounding that enables 2D Vision-Language Models to localize objects in 3D scenes. The framework reformulates 3D data into query-aligned rendered 2D images and structured spatial text, achieving 44.1% [email protected] on ScanRefer and 46.1% overall accuracy on Nr3D, improvements of 7.7% and 7.1% respectively over prior zero-shot methods.
Key insights
- Leveraging 2D Vision-Language Models for 3D tasks is effective when 3D scene information is intelligently translated into query-aligned rendered 2D images combined with structured textual descriptions.
- Integrating fine-grained visual cues from dynamically rendered images with 3D spatial semantics is crucial for precise object localization, particularly in ambiguous or cluttered 3D environments.
- Dynamically selecting viewpoints aligned with the natural language query significantly enhances grounding accuracy, outperforming static rendering strategies, especially for complex or view-dependent references.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.