Reproduce “Input-Aware Sparse Attention for Real-Time Co-Speech Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Beijia Lu, Ziyi Chen, Jing Xiao, Jun-Yan Zhu
Researchers from Carnegie Mellon University and PAII Inc. developed a conditional video distillation framework that uses input-aware sparse attention and input-aware distillation loss to achieve real-time, high-quality co-speech video generation. This approach provides a 13.1x speedup over the teacher model, reaching 25.3 FPS, while improving lip synchronization and hand motion realism.
Key insights
- Leveraging domain-specific structural cues like human pose information can effectively guide attention mechanisms to significantly reduce computational overhead without sacrificing visual fidelity.
- Targeted distillation with region-aware loss functions is crucial for preserving and enhancing perceptual quality in critical areas (face, hands) during model acceleration.
- Combining input-aware sparse attention with input-aware distillation loss enables the creation of efficient, high-fidelity diffusion-based models capable of real-time co-speech video generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.