Reproduce “Leveling3D: Leveling Up 3D Reconstruction with Feed-Forward 3D Gaussian Splatting and Geometry-Aware Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yiming Huang, Baixiang Huang, Beilei Cui, Chi Kit Ng, Long Bai, Hongliang Ren

Leveling3D presents a framework for holistic 3D reconstruction and generation that combines feed-forward 3D Gaussian Splatting with geometry-conditioned diffusion modeling. This approach significantly reduces artifacts and enhances geometric consistency in extrapolated novel views from sparse inputs, outperforming existing diffusion-based methods in image synthesis and depth estimation.

Key insights

  • Injecting explicit 3D geometric priors directly into a diffusion model's latent space through a specialized adapter significantly improves multi-view consistency and realism of generated extrapolated views.
  • A feedback mechanism, where diffusion-enhanced views refine the underlying 3D Gaussian Splatting representation, yields a more accurate and complete 3D scene model than merely performing 2D post-processing.
  • Careful training data curation using palette filtering and adaptive test-time mask refinement are crucial for maintaining structural coherence and preventing artifacts while generating plausible content in under-constrained regions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.