Reproduce “SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinhao Mei, Gael Le Lan, Haohe Liu, Zhaoheng Ni, Varun Nagaraja, Yang Liu, Yangyang Shi, Vikas Chandra
SLAP, a scalable language-audio pretraining framework from Meta, processes variable-duration audio and employs a multi-objective training strategy on 109 million audio-text pairs. The framework establishes new state-of-the-art performance across diverse audio-language benchmarks, including audio-text retrieval, zero-shot audio classification, and audio tagging.
Key insights
- Scaling audio-language pretraining to over 100 million audio-text pairs significantly enhances the robustness and generalizability of learned audio representations.
- Natively handling variable-duration audio inputs up to 30 seconds improves real-world applicability and model performance, especially on datasets with longer clips.
- Combining contrastive, self-supervised, and captioning objectives in a single-stage framework effectively learns both global semantic alignments and fine-grained audio features, yielding a more versatile audio encoder.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.