Reproduce “SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yukai Shi, Weiyu Li, Zihao Wang, Hongyang Li, Xingyu Chen, Ping Tan, Lei Zhang
SceneMaker, developed by researchers from Tsinghua University, HKUST, and IDEA Research, presents a decoupled framework for open-set 3D scene generation from a single image, achieving superior de-occlusion and more accurate 3D object poses. It consistently outperforms state-of-the-art methods across various metrics on challenging indoor and open-set datasets, demonstrating improved generalization.
Key insights
- Decoupling 3D scene generation into de-occlusion, object generation, and pose estimation allows for specialized training on optimal data, improving open-set generalization.
- Rich open-set priors for de-occlusion can be effectively acquired by fine-tuning image editing models on extensive, custom-curated 2D occlusion datasets.
- A diffusion-based pose estimation model with tailored attention mechanisms (local, global, and decoupled cross-attention) can accurately predict 6D poses and sizes for objects in complex, occluded scenes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.