Reproduce “Scaling Speech Tokenizers with Diffusion Autoencoders” on OpenResearch

About this paper

Yuancheng Wang, Zhenyu Tang, Yun Wang, Arthur Hinsvark, Yingru Liu, Yinghao Li, Kainan Peng, Junyi Ao and 4 more

Researchers from Meta Superintelligence Labs and The Chinese University of Hong Kong, Shenzhen introduced SiTok, a Speech Diffusion Tokenizer, designed to learn highly compressed, semantically rich, and high-fidelity speech representations. This model achieves competitive performance across speech understanding, reconstruction, and generation at an ultra-low 0.2 kbps bit-rate, significantly outperforming prior methods.

Key insights

• A generative diffusion autoencoder explicitly models uncertainty introduced by aggressive quantization, enabling high-fidelity reconstruction even at extremely low bit-rates.
• Direct semantic supervision on the discrete latent space using CTC loss is critical for learning linguistically meaningful representations, preventing semantic degradation under compression.
• While larger models improve reconstruction, an optimal model size exists for understanding tasks, suggesting a balance between fine-grained acoustic detail and abstract discriminative features.

Summary by alphaXiv.