Reproduce “DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyu Huang, Yihan Zeng, Zhilu Zhang, Wan Xu, Hang Xu, Songcen Xu, Rynson W. H. Lau, Wangmeng Zuo
DreamControl introduces a two-stage 2D-lifting framework for text-to-3D generation, which first creates a robust 3D self-prior and then uses control-based score distillation to synthesize fine-grained details. This approach effectively mitigates the Janus problem, achieving a low 3D inconsistency rate of 10.00% and a high texture fidelity PickScore of 27.46%, while supporting versatile control for user-guided creation and animation.
Key insights
- Viewpoint bias in 2D diffusion models and overfitting during optimization are fundamental causes of the Janus problem in 2D-lifting text-to-3D generation.
- A coarse yet geometrically consistent 3D shape can be implicitly formed during the early stages of NeRF optimization, serving as a valuable "3D self-prior" to guide further refinement.
- Leveraging control-based guidance with geometric conditions (e.g., edge maps, normal maps) enables robust optimization for consistent and detailed 3D content without sacrificing texture quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.