Reproduce “VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Chen, Yingwei Pan, Haibo Yang, Ting Yao, Tao Mei
VP3D introduces a novel framework for text-to-3D generation that leverages explicit 2D visual prompts to bridge the modality gap between text and 3D content. The method achieves superior visual fidelity and geometric consistency in generated 3D assets by employing a Visual-Prompted Score Distillation Sampling approach, view-dependent visual prompting, and reward-based optimization.
Key insights
- Explicit visual information from a generated 2D image provides more precise guidance for 3D generation than text alone.
- Dual conditioning on both text and visual prompt embeddings in Score Distillation Sampling (VP-SDS) leads to sharper gradients and higher quality 3D outputs.
- Generating and dynamically selecting view-dependent visual prompts helps resolve multi-view inconsistencies and the 'Janus problem' in 3D generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.