Reproduce “M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Masahiro Yasuda, Shunsuke Tsubaki, Keisuke Imoto

M2D-CLAP unifies self-supervised audio representation learning with audio-language alignment, creating a general-purpose model that excels in both transfer learning and zero-shot classification tasks. This approach achieves top average performance in linear evaluation across diverse audio tasks and sets a new state-of-the-art for zero-shot music genre classification on GTZAN.

Key insights

  • Successfully integrating masked prediction (M2D) with contrastive audio-language alignment (CLAP) into a single, unified pre-training framework creates a versatile audio representation.
  • A carefully balanced loss function, primarily focusing on self-supervised learning with language alignment as an auxiliary signal, proves effective for achieving dual capabilities.
  • While offering general-purpose utility, some trade-offs may exist, as seen in slightly lower performance on highly specific, language-agnostic tasks like speaker identification.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.