Reproduce “REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou Yu
REALM, an MLLM-agent framework, integrates multimodal large language models with 3D Gaussian Splatting to perform reasoning-based 3D segmentation and editing. It achieves superior performance on complex, implicit language queries, outperforming prior methods on relevant benchmarks by over 35% mIoU.
Key insights
- Sophisticated 2D reasoning capabilities of MLLMs can be effectively transferred to 3D environments by integrating them with a consistent 3D scene representation and a multi-view grounding strategy.
- A hierarchical Global-to-Local Spatial Grounding approach is crucial for robust 3D object grounding by mitigating viewpoint sensitivity and aggregating diverse MLLM outputs.
- A 3D feature field, optimized for consistent instance identification across multiple views, provides a stable link between 2D MLLM reasoning and specific 3D objects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.