Reproduce “Generalizable Patch-Based Neural Rendering” on OpenResearch

About this paper

Mohammed Suhail, Carlos Esteves, Leonid Sigal, Ameesh Makadia

A neural rendering model achieves state-of-the-art generalizable novel view synthesis by employing a patch-based transformer architecture with canonicalized geometric encodings. It outperforms existing methods in generalization performance and data efficiency on unseen scenes, operating without deep convolutional features or NeRF-like volume rendering.

Key insights

• High-quality, generalizable neural rendering can be achieved using a geometrically-informed, patch-based transformer architecture without relying on deep convolutional features or volumetric rendering.
• Canonicalizing geometric inputs to be invariant to arbitrary scene reference frames is crucial for enabling robust generalization across diverse and unseen scenes.
• Directly blending source pixel colors, guided by attention weights from a multi-stage transformer, proves more effective for generalization than aggregating features in abstract latent spaces.

Summary by alphaXiv.