Reproduce “Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qirui Wu, Denys Iliash, Daniel Ritchie, Manolis Savva, Angel X. Chang
Diorama presents a zero-shot system that reconstructs holistic, compositional, and editable 3D indoor scenes from a single RGB image. It achieves this by leveraging off-the-shelf visual foundation and large multimodal models to parse the image, retrieve CAD models, and optimize their placement, demonstrating strong generalization to diverse real-world and synthetic inputs.
Key insights
- Pre-trained visual foundation models and LMMs can be effectively composed in a modular system to achieve zero-shot, open-world 3D scene perception from a single image.
- Hierarchical multimodal shape retrieval, combining text and image queries, enhances the accuracy of selecting appropriate 3D CAD models from large, diverse datasets.
- A multi-stage, differentiable optimization process is essential for refining object poses and architectural elements to ensure physical plausibility and structural integrity within reconstructed scenes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.