Reproduce “Avatar V: Scaling Video-Reference Avatar Video Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Benjamin Liang, Ce Chen, Desmond Lin, Ivan Somov, Jiajun Zhao, Jiewei Yuan, Jingfeng Zhang, Junhao Huang and 15 more

HeyGen Research developed Avatar V, a system for video-reference avatar video generation that employs a Diffusion Transformer architecture with sparse reference attention and a multi-stage training pipeline. The system generates high-fidelity, behaviorally recognizable avatars, outperforming other methods in identity consistency, lip synchronization, and motion naturalness across objective and subjective evaluations.

Key insights

  • Conditioning avatar generation on full video references, facilitated by sparse attention, effectively captures both static identity and dynamic behavioral patterns, leading to more behaviorally recognizable avatars.
  • Integrating a dedicated motion representation stream within a closed-loop training framework is crucial for unifying appearance and motion style, enhancing the naturalness and authenticity of generated talking avatars.
  • Progressive training, combined with multi-stage distillation and human feedback alignment, is essential for developing production-quality avatar generation systems that are both highly efficient and perceptually aligned with human preferences.
  • Auxiliary losses operating in learned representation spaces can provide semantically meaningful supervision for critical facial features, addressing the sparse signal limitations of standard pixel-level losses.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.