Reproduce “Relit-LiVE: Relight Video by Jointly Learning Environment Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye and 2 more

Researchers from Nanjing University, BAAI, and Tsinghua University developed Relit-LiVE, a framework for video relighting that generates physically consistent and temporally stable relit videos without requiring camera pose knowledge. It achieves this by combining raw RGB information with intrinsic scene properties and jointly learning to produce both relit videos and view-aligned environment videos, resulting in superior visual fidelity and temporal stability compared to baselines on various datasets.

Key insights

  • Complex lighting effects, difficult to capture solely by intrinsic decomposition, can be effectively handled by fusing raw RGB video cues with estimated intrinsic properties during rendering.
  • Video relighting can be reframed as a joint generation task to simultaneously predict the relit video and its corresponding view-aligned environment map, eliminating the need for explicit camera pose information.
  • Enhancing training data diversity through latent-space interpolation and employing a cycle-consistent self-supervised scheme are crucial for achieving robust generalization and temporal stability in relighting real-world videos.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.