Reproduce “Learning to Predict Indoor Illumination from a Single Image” on OpenResearch

About this paper

Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, Jean-François Lalonde

Researchers developed an end-to-end deep learning framework to infer high dynamic range (HDR) illumination, including light source positions and intensities, from a single low dynamic range (LDR) indoor photograph. The method leverages a novel light source detector trained on LDR panoramas and a newly captured HDR dataset to accurately relight virtual objects, with a user study indicating its output is often indistinguishable from ground truth.

Key insights

• Directly mapping LDR image pixels to HDR illumination through an end-to-end deep neural network provides a robust alternative to inverse rendering, avoiding the need for explicit, error-prone intermediate models of geometry or material properties.
• Large existing LDR panorama datasets can be effectively leveraged for initial network training by automatically generating light source annotations and employing a panorama warping operation to simulate localized indoor lighting conditions.
• A custom differentiable cosine filter loss allows for stable and accurate learning of light source positions on a spherical domain by blurring the target mask initially and gradually sharpening it, preventing issues like harsh gradients or spherical harmonic artifacts.
• The network can infer plausible out-of-view light sources and their intensities from limited field-of-view LDR images, demonstrating robust generalization to complex real-world scenarios and diverse input photos.
• Training a robust light source detector and subsequent network on light location first, then fine-tuning for intensity with higher dynamic range data, proves an effective strategy for overcoming data scarcity in HDR illumination estimation.

Summary by alphaXiv.