Reproduce “Z3D: Zero-Shot 3D Visual Grounding from Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikita Drozdov, Andrey Lemeshko, Nikita Gavrilov, Anton Konushin, Danila Rukhovich, Maksim Kolodiazhnyi
Z3D introduces a universal zero-shot 3D visual grounding pipeline that localizes objects in 3D scenes from natural language queries using multi-view images. It establishes new state-of-the-art performance in zero-shot 3DVG, including image-only scenarios, by integrating high-quality 3D instance segmentation and advanced Vision-Language Model (VLM) reasoning, achieving significant improvements on ScanRefer and Nr3D benchmarks.
Key insights
- Combining advanced VLM reasoning for view selection and prompt-guided 2D segmentation with high-quality zero-shot 3D instance segmentation significantly improves zero-shot 3DVG performance.
- Robust 3D reconstruction from multi-view images, even without explicit depth or pose supervision, can effectively enable zero-shot 3DVG in challenging image-only settings.
- A two-stage view selection approach (initial lightweight filtering followed by VLM refinement) efficiently reduces computational load while maintaining accuracy in VLM-based pipelines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.