Reproduce “SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Helin Wang, Jiarui Hai, Yen-Ju Lu, Karan Thakkar, Mounya Elhilali, Najim Dehak

SoloAudio is a generative model for Target Sound Extraction (TSE) that utilizes latent diffusion and a Transformer architecture. It demonstrates improved performance over previous generative models like DPM-TSE and achieves strong zero-shot and few-shot capabilities by leveraging synthetic data. The model records a Purity score of 4.263 0.109 in subjective human evaluations on real-world AudioSet data.

Key insights

  • Applying diffusion directly in the latent space of an audio Variational Autoencoder (VAE), rather than on spectrograms, yields superior audio reconstruction quality.
  • Integrating a Diffusion Transformer (DiT) with long skip connections significantly enhances feature learning and denoising for audio data.
  • Utilizing a CLAP model for conditioning allows for flexible target sound extraction through both audio and natural language queries, enabling zero-shot capabilities.
  • High-quality synthetic audio generated by text-to-audio models can effectively address data scarcity and improve model generalization to unseen sound events.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.