Reproduce “Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wangguandong Zheng, Haifeng Xia, Rui Chen, Ming Shao, Siyu Xia, Zhengming Ding

The Sketch3D framework generates realistic 3D assets from input sketches and text prompts by leveraging 3D Gaussian Splatting and multi-modal guidance, achieving high fidelity and consistency in approximately 3 minutes. This approach demonstrates superior performance over existing baselines in both objective metrics, such as CLIP similarity and SSIM, and subjective user evaluations.

Key insights

  • Initialization of 3D Gaussians using a 3D diffusion model (like Shap-E) provides a more realistic and comprehensive 3D prior compared to Score Distillation Sampling (SDS) from random Gaussians.
  • A distribution transfer mechanism is crucial for structurally optimizing the 3D Gaussians to align rendered content images with multi-view style-consistent guidance images, maintaining shape coherence.
  • Utilizing L2 distance on mid-level CLIP image encoder activations offers a more effective geometric constraint for enforcing sketch similarity than high-level semantic similarity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.