Reproduce “$ϕ$-Scene: Physically Grounded Image-to-3D Scene Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haodong Li, Lulu Shao, Haolin Lu, Yu Fu, Yen-Ru Chen, Seemandhar Jain, Manmohan Chandraker
ϕ-Scene introduces a method for image-to-3D scene reconstruction that generates physically stable 3D scenes coherent with an input image, leveraging Multimodal Large Language Models for complex support topology inference. The approach reduces physical inconsistencies like object interpenetrations and floatings, yielding lower post-simulation translational and rotational drift compared to prior methods.
Key insights
- Incorporating physical grounding functions as an effective structural prior, enhancing both geometric fidelity and scene arrangement accuracy in 3D reconstruction.
- Multimodal Large Language Models can robustly infer complex physical support topologies, including non-vertical and multi-parent relationships, from 2D image data.
- A two-stage sequential physical assembly, involving SDF-based penetration resolution and rigid-body simulation guided by an inferred topology, is essential for achieving both collision-free and dynamically stable scenes that align with the reference image.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.