Reproduce “Towards Practical Real-Time Low-Latency Music Source Separation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junyu Wu, Jie Liu, Tianrui Pan, Jie Tang, Gangshan Wu

The RT-STT model presents a lightweight, real-time, low-latency music source separation system, achieving an average cSDR of 5.17 dB on MUSDB18-HQ. It incorporates perturbation-aware token selection and modality-aware budget allocation for key-value caches, resulting in a model with only 383K parameters and an inference time of 1.01 ms after 16-bit quantization, which is three times faster than previous state-of-the-art real-time methods.

Key insights

  • Feature fusion through channel expansion effectively enhances separation quality by leveraging inter-source dependencies while simultaneously reducing inference time due to better hardware parallelization.
  • Single-path modeling, which focuses exclusively on the time dimension, offers comparable performance to dual-path approaches in real-time, low-latency MSS, but with reduced computational overhead and faster inference.
  • 16-bit Post-Training Quantization can achieve an 82.6% reduction in inference time for audio separation models with minimal impact on separation quality (within 0.01 dB cSDR).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.