Reproduce “Real-time 3D-aware Portrait Video Relighting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Cai, Kaiwen Jiang, Shu-Yu Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, Lin Gao
Researchers from the Chinese Academy of Sciences and collaborating institutions developed a system for real-time 3D-aware relighting and novel view synthesis in portrait videos. This approach distills a generative model into an efficient network, achieving an average inference speed of 32.98 frames per second while maintaining high visual fidelity and temporal consistency.
Key insights
- Effective knowledge distillation from complex 3D-aware generative models into specialized feed-forward networks enables unprecedented real-time performance for video manipulation.
- Maintaining disentangled albedo and shading representations within tri-planes is crucial for robust 3D reconstruction and independent control over appearance and illumination.
- Explicitly modeling temporal coherence with a dedicated network operating on tri-plane features is essential to achieve smooth, artifact-free results in video relighting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.