Reproduce “Stereo Magnification with Multi-Layer Images” on OpenResearch

About this paper

Taras Khakhulin, Denis Korzhenkov, Pavel Solovev, Gleb Sterkin, Timotei Ardelean, Victor Lempitsky

The StereoLayers system learns scene-adapted geometry for a minimal number of semitransparent layers from a stereo image pair, enabling high-quality novel view synthesis. This approach achieves superior rendering quality and real-time performance on mobile devices, consistently outperforming a 32-plane baseline with only 4 layers across various benchmarks.

Key insights

• Learning scene-adapted geometry with a few deformable layers (e.g., 4) significantly outperforms a larger number of fixed, uniformly spaced planes (e.g., 32) for novel view synthesis.
• End-to-end differentiable training of both geometric layer prediction and layer texturing is critical for achieving high-quality results, surpassing methods that perform geometry adaptation as a non-learned post-processing step.
• Incorporating information from both input stereo views (reference and side) during the texturing process is essential for superior rendering quality, especially for view extrapolation.
• The newly introduced SWORD dataset, featuring diverse urban scenes with many close objects and occlusions, provides a more robust and challenging benchmark for evaluating novel view synthesis methods' generalization capabilities.

Summary by alphaXiv.