Reproduce “Generic 3D Diffusion Adapter Using Controlled Multi-View Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hansheng Chen, Ruoxi Shi, Yulin Liu, Bokui Shen, Jiayuan Gu, Gordon Wetzstein, Hao Su, Leonidas Guibas
The MVEdit framework adapts pre-trained 2D diffusion models for high-quality, 3D-consistent multi-view object generation through a training-free 3D adapter. This approach achieves superior 3D alignment, visual quality, and efficiency, generating 3D objects in minutes compared to hours for optimization-based methods, and exhibiting improved metrics such as LPIPS (0.139 vs 0.171) and FID (29.3 vs 42.1) on image-to-3D tasks.
Key insights
- Robust 3D consistency in multi-view diffusion can be achieved by iteratively reconstructing a 3D representation and using its rendered views to condition the 2D diffusion model, effectively guiding it towards geometrically sound outputs.
- Ancestral sampling, when combined with a 3D adapter for controlled multi-view editing, offers a more efficient and diverse 3D generation pipeline compared to optimization-based methods, while maintaining high visual quality.
- Domain-specific 3D initialization can be significantly accelerated and enhanced by fine-tuning 2D diffusion models on orthographic views of 3D objects, enabling efficient knowledge transfer and high-quality generation from smaller datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.