Reproduce “Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Luo, Rongzhi Gu, Jixun Yao
A framework for autoregressive speech generation employs a Locodec tokenizer to create low-frame-rate, high-dimensional continuous tokens and an MP-ELD generative model to ensure stability. This system demonstrated improved single-token predictability, reduced error accumulation in long-form synthesis, and achieved a 0.95% WER on Seed-TTS-eval, without requiring external pretrained models.
Key insights
- Co-designing the representation space and the generative framework is crucial; Locodec's structured high-dimensional continuous latent space significantly improves the generator's predictability and stability.
- Robust generative capabilities, including competitive content alignment and stable long-form synthesis, can be achieved from intrinsic architectural and training objective designs without relying on external pretrained models.
- Multi-path information routing with functionally disentangled conditioning signals and time-dependent CFG schedules effectively mitigates AR error accumulation, allowing precise control over different aspects of generation like content alignment and acoustic consistency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.