Reproduce “Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fei Yu, Quan Deng, Shengeng Tang, Yuehua Li, Lechao Cheng
Researchers from Liaoning University of Technology, University of the Chinese Academy of Sciences, Hefei University of Technology, and Zhejiang Lab developed OSU-3DSG, an annotation-free framework that integrates Vision-Language Models for dynamic 3D scene graph generation and retrieval-augmented reasoning. This system achieved competitive to superior performance in open-vocabulary 3D scene graph generation and interaction tasks, notably surpassing existing methods in predicate prediction and task planning metrics without supervised training.
Key insights
- Leveraging powerful 2D Vision-Language Models (VLMs) combined with optimal viewpoint selection significantly enhances open-vocabulary object grounding and relation inference in dynamic 3D scenes without requiring direct 3D supervision.
- Transforming the constructed 3D scene graph into a vector-searchable knowledge base enables retrieval-augmented reasoning with Large Language Models (LLMs), facilitating robust and context-aware multimodal interaction.
- Strategic filtering of object pairs based on geometric and spatial priors (e.g., 3D IoU and Euclidean distance) is essential for efficiently extracting meaningful relationships and improving predicate prediction accuracy in 3D scene graph generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.