Reproduce “Analysing the Language of Neural Audio Codecs” on OpenResearch

About this paper

Joonyong Park, Shinnosuke Takamichi, David M. Chan, Shunsuke Kando, Yuki Saito, Hiroshi Saruwatari

This study analyzes the statistical and linguistic properties of discrete speech tokens generated by Neural Audio Codec (NAC) models, comparing them to natural language and correlating these properties with speech quality. The research finds that NAC tokens, particularly at the 3-gram level, exhibit statistical regularities consistent with Zipf's and Heaps' laws, and that tokens adhering more closely to these linguistic patterns achieve improved intelligibility and naturalness in resynthesized speech.

Key insights

• NAC token sequences, particularly at the 3-gram level, consistently exhibit statistical properties that align with established linguistic laws like Zipf's law (best fit with $\alpha$ approaching 2) and Heaps' law (high similarity in $K$ and $\beta$ exponents to natural language).
• There is a strong correlation between these linguistic statistical properties and downstream speech performance: tokens with stronger adherence to Zipf's law, more consistent vocabulary growth, and lower redundancy are associated with improved speech intelligibility and naturalness.
• The 3-gram granularity appears to be an optimal balance for NAC tokens, effectively capturing local linguistic patterns while maintaining sufficient token diversity, which contrasts with 1-gram (less reuse) and higher n-grams (high sparsity).

Summary by alphaXiv.