Reproduce “Towards Learning Universal Audio Representations” on OpenResearch

About this paper

Luyu Wang, Pauline Luc, Yan Wu, Adria Recasens, Lucas Smaira, Andrew Brock, Andrew Jaegle, Jean-Baptiste Alayrac and 3 more

DeepMind researchers developed HARES, a new meta-benchmark for evaluating universal audio representations, and introduced Slowfast NFNet-F0, a novel architecture that achieves state-of-the-art performance across diverse audio domains including speech, music, and environmental sounds. The Slowfast NFNet-F0, trained with a SimCLR self-supervised objective, significantly improved scores on the HARES benchmark, particularly in music and cross-domain speech tasks.

Key insights

• Self-supervised contrastive learning (SimCLR) consistently yields better generalization across speech and music domains compared to supervised classification pre-training, highlighting its effectiveness for learning broadly applicable audio features.
• The Slowfast architecture's ability to process audio spectrograms with two pathways (slow for high-level semantics, fast for rapid temporal changes) and careful selection of normalization schemes (LayerNorm, InstanceNorm, or normalizer-free) are crucial for robust cross-domain transferability.
• Domain-specific models, while strong in their target area, exhibit poor transferability to other audio domains, underscoring the need for a holistic approach to representation learning.

Summary by alphaXiv.