Reproduce “IP-FaceDiff: Identity-Preserving Facial Video Editing with Diffusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tharun Anand, Aryan Garg, Kaushik Mitra

IP-FaceDiff introduces a framework for text-driven facial video editing using diffusion models, demonstrating approximately 80% faster inference times than existing benchmarks. It effectively preserves facial identity, maintains temporal consistency across frames, and enables precise, localized edits, even in challenging "in-the-wild" videos.

Key insights

  • Separately fine-tuning a pre-trained diffusion model for identity preservation and localized editing, using L1/ArcFace loss and directional CLIP loss respectively, allows for robust control over both aspects.
  • Integrating identity guidance during inference by leveraging precomputed self-attention features from an identity-preserving model is crucial for maintaining facial identity and head pose across edits.
  • Combining a keyframe-based editing approach with latent token propagation provides substantial efficiency gains and improved temporal consistency compared to methods that process frames individually.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.