Reproduce “NeX: Real-time View Synthesis with Neural Basis Expansion” on OpenResearch

About this paper

Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, Supasorn Suwajanakorn

The NeX system introduces a novel multiplane image representation that incorporates neural basis expansion to model view-dependent effects, enabling real-time novel view synthesis with photorealistic rendering quality. This approach achieves over 300 frames per second on an NVIDIA RTX 2080Ti, accurately reproducing complex reflections and refractions while outperforming state-of-the-art methods like NeRF in both speed and fidelity.

Key insights

• Leveraging learned, scene-specific basis functions for view-dependent color representation is more effective than using fixed basis functions (e.g., spherical harmonics) for compact and accurate modeling.
• Explicitly optimizing a base color component (K0) alongside implicit neural predictions for view-dependent effects significantly improves the recovery of fine details and high-frequency textures.
• Combining the explicit structure of MPIs with implicit neural modeling of view-dependent components offers a powerful balance between rendering speed and photorealistic quality.

Summary by alphaXiv.