Reproduce “ERUPT: Efficient Rendering with Unposed Patch Transformer” on OpenResearch

About this paper

Maxim V. Shugaev, Vincent Chen, Maxim Karrenbach, Kyle Ashley, Bridget Kennedy, Naresh P. Cuntoor

ERUPT (Efficient Rendering with Unposed Patch Transformer) proposes a novel view synthesis model that reconstructs diverse scenes from small collections of unposed RGB images, achieving rendering speeds exceeding 600 frames per second for 224x224 images and improved perceptual quality with generative models. Researchers also released MSVS-1M, a new large-scale real-world benchmark dataset for generalized latent view synthesis.

Key insights

• Patch-based decoding significantly improves computational efficiency (speed and VRAM) for novel view synthesis, reducing rendering and training times by an order of magnitude compared to pixel-based methods.
• A "Target Camera Switching" training strategy allows a single model to learn from both posed and unposed imagery, enabling robust reconstruction from limited real-world data while maintaining direct camera control at inference.
• Leveraging pre-trained vision transformer backbones (like DINOv2) for feature extraction, combined with LoRA fine-tuning, enhances generalization and performance in 3D scene synthesis tasks.

Summary by alphaXiv.