Reproduce “Scaling up masked audio encoder learning for general audio classification” on OpenResearch

About this paper

Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, Bin Wang

Researchers at Xiaomi Corporation's AI Lab introduce Dasheng, a self-supervised audio encoder, by scaling masked autoencoder (MAE) learning to 1.2 billion parameters and pretraining on over 272,000 hours of diverse audio data. This approach establishes a general audio backbone that achieves an overall average HEAR benchmark score of 81.25%, outperforming all other compared models, and demonstrates significant improvements in cross-domain generalization and k-nearest neighbor classification on various speech, music, and environmental sound tasks.

Key insights

• Scaling both model size and dataset size for MAE-based audio encoders leads to substantial improvements in performance and generalization across diverse audio domains (speech, music, environmental sounds).
• The inherent computational efficiency of the MAE framework, by processing only unmasked data during encoding, makes large-scale pretraining feasible for audio representation learning.
• The resulting Dasheng embeddings capture inherently rich and generalizable audio information, enabling strong performance even with simple k-nearest neighbor classification without further fine-tuning.

Summary by alphaXiv.