Reproduce “ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Huiqi Wu, Jianbo Mei, Yingjie Huang, Yining Xu, Jingjiao You, Yilong Liu, Li Yao

The ESCT3D framework generates 3D content from text by combining GPT-4V-driven prompt self-optimization with Gaussian Splatting, enhanced by extensive multi-view constraints. This approach yields richer, more consistent 3D models with selective control and achieves superior CLIP scores (e.g., ViT-bigG-14: 49.90) in 10 minutes, addressing limitations in prompt engineering and multi-view fidelity.

Key insights

  • Leveraging an LLM like GPT-4V for iterative prompt self-optimization can effectively bridge the gap between simple user input and the detailed prompts required for high-quality generative models.
  • The strategic integration of extensive multi-view geometric and feature-level constraints during 3D Gaussian Splatting optimization is critical for overcoming the "Janus problem" and enhancing detail consistency across diverse viewpoints.
  • Combining efficient 3D representations like Gaussian Splatting with sophisticated 2D diffusion priors and multi-modal conditional inputs enables both faster generation and more precise control over 3D asset creation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.