Reproduce “Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, Daniel Cohen-Or

Researchers at Tel Aviv University developed Latent-NeRF, a framework that generates 3D shapes and textures from text prompts with improved efficiency and explicit user control. The method leverages Latent Diffusion Models (LDMs) by having NeRFs operate directly in the LDM's latent space, reducing generation time to under 15 minutes, and introduces shape guidance (Sketch-Shape) and texture painting (Latent-Paint) capabilities.

Key insights

  • Directly rendering NeRFs into the latent space of Latent Diffusion Models significantly improves computational efficiency by avoiding repeated encoding/decoding steps.
  • Lenient geometric constraints, rather than strict adherence, are effective for guiding 3D generation, allowing for user control over coarse shape while maintaining creative freedom for detailed text-guided synthesis.
  • Pre-trained 2D diffusion models can be effectively leveraged for high-quality, text-conditional texture generation on 3D meshes by operating within their latent feature spaces.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.