Reproduce “SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuefei Sun, Xujia Zhang, Brendan Crowe, Doncey Albin, Christoffer Heckman
SceneGraphGrounder proposes a framework for zero-shot 3D visual grounding by treating it as a structured graph matching problem between a query graph from natural language and a reconstructed 3D scene graph. The system achieved 36.8% [email protected] and 26.0% [email protected] on the ScanRefer benchmark using only RGB-D inputs and odometry, and demonstrated a 51.2% success rate in real-world robot experiments.
Key insights
- Explicitly structuring 3D scenes and language queries as graphs allows for robust and interpretable relational reasoning, overcoming limitations of implicit representations.
- Leveraging 2D visual context via a marker-guided VLM prompting strategy effectively extracts and lifts rich spatial relationships into a persistent 3D scene graph.
- Combining structured graph matching with a VLM-based disambiguation module provides synergistic benefits, where graph matching handles clear relational queries and the VLM resolves ambiguous cases using broader visual context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.