Reproduce “Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Philippe Esling, Axel Chemla--Romeu-Santos, Adrien Bitton
A framework is presented for creating generative timbre spaces by regularizing Variational Auto-Encoders with human perceptual dissimilarity ratings. This approach enables perceptually meaningful timbre synthesis and analysis, allowing for smooth morphing between instruments and coherent embedding of novel sounds.
Key insights
- Perceptual regularization effectively shapes the VAE's latent space to mirror human timbre dissimilarity judgments, crucially without degrading audio reconstruction quality.
- The Non-Stationary Gabor Transform (NSGT) provides a superior input representation, enabling better reconstruction and more effective perceptual alignment compared to STFT or DCT.
- Audio descriptors like spectral centroid and bandwidth display locally smooth but globally non-linear relationships across the learned latent space, offering deeper insights into timbre perception.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.