Reproduce “Acoustic BPE for Speech Generation with Discrete Tokens” on OpenResearch

About this paper

Feiyu Shen, Yiwei Guo, Chenpeng Du, Xie Chen, Kai Yu

This research from Shanghai Jiao Tong University introduces Acoustic Byte-Pair Encoding (aBPE) to compress discrete audio tokens for speech generation, demonstrating up to a 5x inference speedup and enhanced linguistic coherence. The method also enables an effective acoustic rescoring approach to select optimal synthetic speech from diverse Text-to-Speech outputs, aligning well with human preferences.

Key insights

• Acoustic BPE significantly compresses discrete audio token sequences, leading to substantial reductions in sequence length and accelerating autoregressive speech generation.
• The use of aBPE enhances the SLM's capacity to learn and capture underlying syntactic structures in speech, improving the linguistic quality of generated audio.
• aBPE-trained SLMs generate speech with improved diversity and richness, suggesting a more complex and varied acoustic output compared to models without aBPE.
• The SLM's probability estimations for aBPE token sequences serve as an effective metric for identifying high-quality synthetic speech, aligning closely with human preferences for naturalness.

Summary by alphaXiv.