Reproduce “Separate What You Describe: Language-Queried Audio Source Separation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D. Plumbley, Wenwu Wang
Researchers at the University of Surrey and ByteDance introduce Language-Queried Audio Source Separation (LASS), a new task and an end-to-end deep learning framework called LASS-Net. LASS-Net effectively separates target audio sources from mixtures using natural language descriptions as queries, showing performance improvements of 0.49 dB SDR over tag-based methods and robust generalization to diverse linguistic inputs.
Key insights
- Natural language descriptions provide richer, more expressive, and more effective query information for audio source separation compared to discrete sound event tags.
- Integrating pre-trained large language models (like BERT) for query encoding with a conditioned audio separation network (via FiLm layers) successfully bridges semantic language understanding with low-level audio processing.
- A jointly trained architecture can effectively learn to extract target sources specified by diverse natural language inputs, demonstrating generalization beyond specific training captions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.