Reproduce “Learning-Based View Synthesis for Light Field Cameras” on OpenResearch

About this paper

Nima Khademi Kalantari, Ting-Chun Wang, Ravi Ramamoorthi

Researchers from UC San Diego and UC Berkeley developed a learning-based method for synthesizing high-quality novel views from sparse light field camera inputs, achieving an average PSNR of 37.50 dB and SSIM of 0.970, which are superior to state-of-the-art techniques. This approach utilizes an end-to-end trained convolutional neural network architecture that first estimates disparity optimized for synthesis, then predicts color, effectively handling occlusions and reducing artifacts prevalent in consumer light field cameras.

Key insights

• Decomposing the view synthesis task into a disparity estimation followed by color prediction is more effective than a single end-to-end network, leading to sharper results.
• Optimizing disparity estimation specifically for the quality of the synthesized image, rather than for geometric accuracy alone, significantly improves rendering fidelity, particularly at occlusion boundaries.
• The Color Predictor CNN learns to intelligently aggregate and utilize information from multiple warped input views and the estimated disparity to reconstruct occluded regions effectively.

Summary by alphaXiv.