Reproduce “Understanding Pure CLIP Guidance for Voxel Grid NeRF Models” on OpenResearch

About this paper

Han-Hung Lee, Angel X. Chang

Researchers at Simon Fraser University and Amii developed a framework for pure CLIP-guided 3D object generation using voxel grids, achieving improved geometric coherency and significantly faster training times compared to prior methods like Dream Fields, while systematically evaluating augmentation and regularization techniques.

Key insights

• A spherical prior KL divergence loss is crucial for the emergence of object densities and overall coherency, especially when augmentations are active, acting as an inductive bias for object centering.
• Implicit voxel grids (Vox_Imp), which use MLPs to predict density and color from feature grids, consistently produce superior geometrical structure and coherency compared to explicit grids, demonstrating the regularization benefits of neural networks.
• Ensembling a smaller CLIP model (e.g., ViT-B/32) with a larger one (ViT-L/14) helps regularize the larger model, preventing adversarial generations and leveraging its capacity for finer details, as larger models alone can be susceptible to noise.

Summary by alphaXiv.