Reproduce “Efficient Vocal Source Separation Through Windowed Sink Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Christodoulos Benetatos, Yongyi Zang, Randal Leistikow
Researchers at University of Rochester and Smule Labs developed Windowed Sink Attention (WSA), an efficient attention mechanism for vocal source separation. The WSA model achieved a 44.5x reduction in attention computations while retaining 92% of the original MelBand-Roformer's Signal-to-Distortion Ratio performance.
Key insights
- Temporal attention in leading vocal separation models is highly localized, with significant attention weights concentrated near the diagonal, suggesting that full self-attention may be computationally redundant for many temporal dependencies.
- A combination of local windowed attention and a small set of global sink tokens can effectively approximate full self-attention, significantly reducing computational complexity from quadratic to linear scaling.
- Targeted optimization of attention mechanisms based on observed data characteristics and inductive biases (e.g., localized temporal dependencies in audio) leads to more efficient and scalable model architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.