Reproduce “Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengchuan Gao, Teng Hu, Bohao Feng, Luchen Li, Wenqiang Wang, Hongqian Deng, Ran Yi
This research introduces a training-free framework to accelerate unified audio-visual generation models by using synchrony-aware protected sparse attention and a novel cache-reuse mechanism. It leverages bidirectional cross-modal attention patterns to identify and preserve synchronization-critical computations, achieving up to 2x end-to-end acceleration on models like Ovi and LTX-2.3 while maintaining or enhancing generation quality and audio-video synchronization.
Key insights
- Bidirectional cross-modal attention naturally provides an intrinsic and reusable saliency signal that accurately identifies tokens vital for maintaining audio-video synchronization.
- Protecting a small fraction (e.g., 5%) of these synchronization-critical tokens during sparse attention and feature caching effectively preserves generation quality and cross-modal alignment.
- The relative stability of cross-modal interaction patterns across nearby denoising steps enables an efficient lagged saliency reuse strategy, minimizing overhead while adapting to changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.