Reproduce “Language-Codec: Bridging Discrete Codec Representations and Speech Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
Researchers at Zhejiang University developed Language-Codec, a discrete acoustic codec designed to improve compatibility with speech language models by controlling information distribution across codebook channels. The model achieved state-of-the-art audio reconstruction quality with fewer channels and improved speaker similarity by 10-15% in zero-shot text-to-speech tasks.
Key insights
- Controlling the information density in the initial codebook channels, as achieved by MCRVQ, significantly improves the suitability of discrete representations for text-to-speech generation by SLMs.
- A decoder architecture that uses Fourier transform-based upsampling can achieve high-fidelity audio reconstruction while effectively handling complex waveform details and reducing artifacts.
- It is possible to achieve high audio reconstruction quality and improve downstream generative tasks with a reduced number of codebook channels, thereby lowering the computational burden on speech language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.