Reproduce “XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, Francis Williams

XCube introduces a generative model capable of producing high-resolution, sparse 3D voxel grids with arbitrary attributes, achieving an effective resolution of up to 1024^3 for large outdoor scenes in under 30 seconds. The model achieves state-of-the-art performance in 3D object and scene generation, supporting applications like user-guided editing and scene completion.

Key insights

  • Generative models operating purely within the 3D domain can achieve high resolutions (up to 1024^3) and scale to complex, expansive scenes without relying on computationally expensive 2D distillation.
  • Combining a hierarchical latent diffusion model with an optimized sparse volumetric data structure (VDB) enables rapid, feed-forward 3D generation of millions of voxels in under 30 seconds.
  • Progressive pruning within the Sparse Structure VAE is critical for preserving fine shape details, introducing 3D inductive bias, and ensuring computational efficiency in hierarchical sparse voxel generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.