Reproduce “Multi3DRefer: Grounding Text Description to Multiple 3D Objects” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Zhang, ZeMing Gong, Angel X. Chang
Researchers from Simon Fraser University and Amii developed Multi3DRefer, a dataset and model for 3D visual grounding that grounds zero, one, or many objects from natural language descriptions. The M3DRef-CLIP model achieves an [email protected] of 38.4% overall on the new task and established state-of-the-art on the ScanRefer benchmark, reaching 45.5% overall accuracy.
Key insights
- Integrating powerful 2D vision-language representations from pre-trained models like CLIP into a 3D grounding pipeline through online rendering substantially boosts performance.
- A symmetric contrastive loss function effectively aids in learning robust multi-modal embeddings, particularly for tasks involving a flexible number of target objects.
- Augmenting datasets with large language models like ChatGPT, followed by rigorous human verification, can generate diverse and natural language descriptions for complex 3D tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.