Reproduce “Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yupeng Zhou, Lianghua Huang, Zhifan Wu, Jiabao Wang, Yupeng Shi, Biao Jiang, Daquan Zhou, Yu Liu and 2 more
Mutual Forcing presents a dual-mode, self-evolving framework for fast autoregressive audio-video character generation, addressing the challenges of joint modeling and rapid long-horizon consistency. It achieves competitive or superior audio-video synchronization and quality with 4-8 sampling steps, significantly outperforming baselines requiring 50-100 steps and enabling real-time generation on a single GPU.
Key insights
- A single, weight-shared model can effectively learn both fast and high-quality generation modes through a self-evolution process, removing the reliance on external teacher models and reducing training overhead.
- The hybrid distillation approach, combining ShortCut and Distribution Matching Distillation components for the few-step mode, provides complementary supervision that leads to improved audio quality and clearer visual details.
- Utilizing self-evolving streaming context and a more balanced temporal attention distribution effectively mitigates exposure bias and prevents quality degradation during long-duration autoregressive generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.