Reproduce “GenLit: Reformulating Single-Image Relighting as Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shrisha Bharadwaj, Haiwen Feng, Giorgio Becherini, Victoria Fernandez Abrevaya, Michael J. Black
GenLit reformulates single-image relighting as a controllable video generation task, leveraging a fine-tuned video diffusion model to dynamically adjust lighting from a moving point source. This approach achieves superior quantitative performance on synthetic data (e.g., RMSE 0.0309, LPIPS 0.0209) and generalizes effectively to real-world images, generating plausible shadows, reflections, and indirect lighting effects.
Key insights
- Video foundation models, trained on extensive image and video datasets, implicitly learn a robust understanding of 3D scenes and how light interacts with materials.
- By modeling relighting as a temporal change in illumination, the inherent capabilities of video diffusion models to synthesize consistent changes over time can be exploited.
- Training on a carefully constructed synthetic dataset featuring dynamic lighting allows the model to generalize effectively to real-world images without requiring explicit 3D reconstruction.
- The dynamic change from ambient to point light source, alongside its movement, is crucial for convincing relighting and the synthesis of realistic shadow details.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.