Reproduce “Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chunyu Li, Jiaye Li, Ruiqiao Mei, Haoyuan Xia, Hao Zhu, Jingdong Wang, Siyu Zhu

The Hallo-Live framework enables real-time, text-driven joint audio-video avatar generation, achieving 20.38 Frames Per Second (FPS) with 0.94 seconds latency, while maintaining high visual fidelity and audio-visual synchronization comparable to non-real-time state-of-the-art models.

Key insights

  • Providing a limited, asynchronous 'look-ahead' audio context to the video stream is critical for achieving natural anticipatory lip movements and timestamp-level audio-video synchronization in real-time streaming.
  • Aggressive distillation can be effectively mitigated by guiding the student model with human-centric multi-modal preferences, which allows the student to potentially surpass the teacher's average quality in specific attributes.
  • A balanced combination of reward signals from visual quality, speech naturalness, and audio-visual synchronization is crucial for robust and high-quality accelerated generation, as individual rewards or excessive weighting can lead to narrow optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.