Reproduce “Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nathan Paek, Yongyi Zang, Qihui Yang, Randal Leistikow

Researchers from Stanford University, UCSD, and Smule Labs developed a framework that uses Sparse Autoencoders (SAEs) and linear probes to map features in audio latent spaces to human-interpretable acoustic concepts like pitch, amplitude, and timbre. This approach facilitates targeted manipulation of generated audio and revealed a coarse-to-fine synthesis hierarchy in models like DiffRhythm, where pitch converges earliest in the generation process.

Key insights

  • Sparse Autoencoders can extract perceptually relevant and disentangled features from diverse audio latent spaces, including continuous VAE and discrete codebook representations.
  • A clear hierarchy of linear decodability for acoustic concepts was observed, with pitch being most linearly separable (75-87% accuracy), followed by loudness (17-63%), and timbre proving most challenging (17-46%).
  • Generative models like DiffRhythm synthesize audio in a coarse-to-fine manner, establishing fundamental properties such as pitch early in the generation process, followed by timbre, and then refining loudness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.