Reproduce “IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation” on OpenResearch

About this paper

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

Researchers from University of Oxford, NEC Labs America, UC Merced, and Google DeepMind present IllumiCraft, a diffusion transformer framework that unifies 3D geometry tracking with HDR illumination control for video relighting, achieving 43% FVD reduction over Light-A-Video baseline and 6x faster inference (105 vs 645 seconds for 49-frame videos) through joint modeling of lighting and geometric cues within a Wan2.1-based architecture, while introducing IllumiPipe data collection pipeline that generates 20,170 video pairs enriched with synchronized HDR maps, 3D point tracks, and relit sequences to address the critical limitation of prior video relighting methods that omit explicit geometric guidance and suffer from lighting inconsistencies when scene geometry changes.

Key insights

• Joint geometry-illumination modeling achieves 18% FVD reduction compared to illumination-only approach, confirming geometry's critical role in spatially-aware light-scene interaction
• Strategic data augmentation with 50% HDR dropout and 30% 3D tracking dropout during training enables robust generalization and text-only control without explicit geometric inputs
• Temporal consistency of HDR maps achieved by warping initial chrome ball extraction across frames using estimated 3D camera motion rather than per-frame extraction
• Explicit geometric guidance addresses fundamental limitation where lighting changes must respect evolving scene structure for physical plausibility

Summary by alphaXiv.